Original: T HAT E SCALATED Q UICKLY : C OMPOUNDING C OMPLEXITY BY E DITING L EVELS AT THE F RONTIER OF A GENT C APABILITIES
Revised: T HAT E SCALATED Q UICKLY : C OMPOUNDING C OMPLEXITY BY E DITING L EVELS AT THE F RONTIER OF A GENT C APABILITIES

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Given a well-speciﬁed task, and enough samples, reinforcement learning (RL) algorithms can produce agents that achieve a high reward.
Revised: Deep Reinforcement Learning (RL) has recently produced impressive results in a series of settings such as games and robotics.

Original: However, a key challenge that limits the utility of RL agents for real-world problems is the agent’ssubsequent ability to generalize to unseen variations (or levels).
Revised: However, a key challenge that limits the utility of RL agents for real-world problems is the agent’s ability to generalize to unseen variations (or levels ).

Original: To train more robust agents, the ﬁeld of Unsupervised Environment Design (UED) seeks to update both the agent and the distribution over environments, producing a curriculum of increasingly complex levels.
Revised: To train more robust agents, the ﬁeld of Unsupervised Environment Design (UED) seeks to produce a curriculum by updating both the agent and the distribution over training environments.

Original: Recent progress in UED led to  agents capable of systematic generalization within an environment, including transfer to  human-designed levels, while also achieving theoretical convergence guarantees.
Revised: Recent advances in UED have come from promoting levels with high regret , which provides theoretical guarantees in equilibrium and empirically has been shown to produce agents capable of zero-shot transfer to unseen human-designed environments.

Original: However,  learning environment adversaries remains a challenge, while prominent replay-based methods rely on the ability of a random generator to produce a sufﬁcient number of levels with high learning potential.
Revised: However, current methods require either learning an environment-generating adversary, which remains a challenging optimization problem, or curating a curriculum from randomly sampled levels, which is ineffective if the search space is too large.

Original: In this paper we instead propose to evolve a curriculum, by making edits to previously useful levels.
Revised: In this paper we instead propose to evolve a curriculum, by making edits to previously selected levels.

Original: Our approach, which we call 
Revised: Our approach, which we call Adversarially Compounding Complexity by

Original: Adversarially Compounding Complexity by Editing Levels (ACCEL),consistently producesincrementally challenging levels at the frontier of an agent’s capabilities, resulting in curricula that start simple but become increasingly complex, leading to agents that can transfer to challenging out-of-distribution environments.
Revised: Editing Levels (ACCEL), produces levels at the frontier of an agent’s capabilities, resulting in curricula that start simple but become increasingly complex. ACCEL maintains the theoretical beneﬁts of prior works, while outperforming them empirically when transferring to complex out-of-distribution environments.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Reinforcement learning (RL, Sutton & Barto (1998)) considers the problem of an agent learning from experience in an environment to maximize total (discounted) of reward.
Revised: Reinforcement Learning (RL, Sutton & Barto (1998)) considers the problem of an agent learning from experience in an environment to maximize total (discounted) of reward.

Original: The past decade has seen a surge of interest in RL, with high proﬁle successes in games (Vinyals et al., 2019; Berner et al., 2019; Silver et al., 2016; Mnih et al., 2013) and robotics (OpenAI et al., 2019; Andrychowicz et al., 2020).
Revised: The past decade has seen a surge of interest in RL, with high proﬁle successes in games (Vinyals et al., 2019; Berner et al., 2019; Silver et al., 2016; Mnih et al., 2013; Hu & Foerster, 2020) and robotics (OpenAI et al., 2019; Andrychowicz et al., 2020).

Original: As such, there is tremendous excitement that RL may be a path towards generally capable agents (Silver et al., 2021).
Revised: As such, there is tremendous excitement that RL may be a path towards generally capable agents (Silver et al., 2021).

Original: However, despite these impressive results, deploying RL agents in the real world remains a challenge (Dulac-Arnold et al., 2019).
Revised: Despite these successes, deploying RL agents in the real world remains a challenge (Dulac-Arnold et al., 2019).

Original: Notably, strong training performance in simulation may not result in policies that are robust to the many sources of variation in the real world.
Revised: Notably, strong training performance in simulation may not result in policies that are robust to the many sources of variation in the real world.

Original: Addressing this challenge on the agent side has become an active area of research (Zhang et al., 2021a; Agarwal et al., 2021a; Raileanu & Fergus, 2021), but in this paper we instead focus on the impact of the training environment itself, which often has a signiﬁcant impact on agent performance (Co-Reyes et al., 2020).
Revised: Addressing this challenge on the agent side has become an active area of research (Zhang et al., 2021a; Agarwal et al., 2021a; Raileanu & Fergus, 2021), but in this paper we instead focus on the impact of the training environment itself, which often has a signiﬁcant impact on agent’s ability to generalize (Co-Reyes et al., 2020).

Original: For example in locomotion tasks, Reda et al.
Revised: For example in locomotion tasks, Reda et al.

Original: (2020) found that the initial state distribution, survival bonus, reward structure and control frequency had a signiﬁcant impact on the performance of an agent.
Revised: (2020) found that the initial state distribution, survival bonus, reward structure and control frequency had a signiﬁcant impact on the performance of an agent.

Original: Indeed, curricula over environments can also inﬂuence the generalization performance of the agent Jiang et al. (2021b).
Revised: Indeed, curricula over environments can also inﬂuence the generalization performance of the agent (Jiang et al., 2021b).

Original: Throughout this paper we consider distributions of environments, referring to each individual sample as a level .
Revised: Throughout this paper we consider distributions of environments, referring to each individual sample as a level .

Original: Given a parameterized environment, the simplest approach one can consider is Domain Randomization (DR, Jakobi, 1997; Tobin et al., 2017; Sadeghi & Levine, 2017; Risi & Togelius, 2020; Peng et al., 2017), whereby an agent trains on individual levels uniformly sampled from an underlying environment distribution.
Revised: Given a parameterized environment, the simplest approach one can consider is Domain Randomization (DR, Jakobi, 1997; Tobin et al., 2017; Sadeghi & Levine, 2017; Risi & Togelius, 2020; Peng et al., 2017), whereby an agent trains on individual levels uniformly sampled from an underlying environment distribution.

Original: It has been shown that training an agent with a DR-type approach can produce agents capable of complex real-world skills (OpenAI et al., 2019).
Revised: It has been shown that training an agent with a DR-type approach can produce agents capable of complex real-world skills (OpenAI et al., 2019).

Original: However, the performance of DR is only as good as the sampling distribution available—thus it can be ineffective when the probability of sampling useful levels is  low.
Revised: However, the performance of DR is only as good as the sampling distribution available—thus it can be ineffective when the probability of sampling useful levels is too low.

Original: Recently,the ﬁeld of Unsupervised Environment Design (UED, Dennis et al., 2020) has emerged as a general approach to generating effective curricula.
Revised: Recently, Unsupervised Environment Design (UED, Dennis et al., 2020) has emerged as formalism for methods to design effective curricula.

Original: Given a parameterized environment, UED methods frame learning as a game between a teacher which generates a curriculum of environments, and a studentagent seeking to maximize some notion of return.
Revised: Given a parameterized environment, UED methods frame learning as a game between a teacher which generates a curriculum of levels, and a student seeking to maximize some notion of return.

Original: UED is a generalization of several other approaches, indeed, DR can be considered as a UED algorithm whereby the teacher generates environments uniformly at random from the environment distribution.
Revised: UED is a generalization of several other approaches. Indeed, DR can be considered as a UED algorithm whereby the teacher generates environments uniformly at random from the environment distribution.

Original: Other approaches to UED consider learning a teacher agent (or generator), with a variety of adversarial objectives proposed (Dennis et al., 2020;
Revised: Other approaches to UED consider learning a teacher agent (or generator), with a variety of adversarial objectives proposed (Dennis et al., 2020;

Original: Gur et al., 2021).
Revised: Gur et al., 2021).

Original: However, it is often challenging to learn the generator, given the nonstationarityof the problem and  the complexity of building entire environments with a sparse reward signal.
Revised: However, training a teacher is a challenging optimization problem, suffering from both nonstationarity and sparse reward, as the teacher’s feedback only comes after evaluation by a changing student policy.

Original: Recent work showed it can be more effective to simply curate levels produced by 
Revised: Recent work showed it can be more effective to simply curate levels produced by DR (Jiang et al.

Original: DR (Jiang et al., 2021b;a; Matiisen et al., 2020), producing a curriculum of increasingly complex randomly generated levels.
Revised: , 2021b;a; Matiisen et al., 2020), producing a curriculum of increasingly complex randomly generated levels.

Original: Despite the promise, these methods can only be as effective as the  random levels they sample, which can be a limitation in high dimensional design spaces.
Revised: Despite their promise, these methods can only be as effective as the best of the random levels they sample, which can be a limitation in high dimensional design spaces.

Original: Finally, another series of promising works seek to evolve populations of environments (Wang et al., 2019; 2020; Dharna et al., 2020), but these methods heavily rely on handcrafted heuristics while also using up to 20x more compute than training a single agent.
Revised: Finally, another series of promising works seek to evolve populations of environments (Wang et al., 2019; 2020; Dharna et al., 2020), but these methods heavily rely on handcrafted heuristics and also use up to 20x more compute since they also train a population of agents.

Original: In this paper we seek a general method harnessing the beneﬁts of evolution to train a single agent.
Revised: In this paper, we seek a general method which harnesses the beneﬁts of all three of these approaches.

Original: We posit the following: Rather than generate levels from scratch, it may be more effective to edit previously curated levels 
Revised: We posit the following: Rather than generate levels from scratch , it may be more effective to edit previously curated levels .

Original: Our primary contribution is to propose a new method which we call Adversarially Compounding
Revised: Our primary contribution is to propose a new method which we call Adversarially Compounding

Original: Complexity by Editing Levels , or ACCEL.
Revised: Complexity by Editing Levels , or ACCEL.

Original: ACCEL can be seen as an evolutionary process, with levels constantly changing to remain at the frontier of the student agent’s capabilities (see: Fig. 2).
Revised: ACCEL is an evolutionary process, with levels constantly changing to remain at the frontier of the student agent’s capabilities (see: Figure 2).

Original: [Figure]
Revised: [Figure]

Original: As such, an emergent property of the levels generated is that they begin simple but quickly become more complex.
Revised: As such, an emergent property of the levels generated is that they begin simple but quickly become more complex.

Original: This beneﬁts both the beginning of training (Berthouze & Lungarella, 2004), as the student begins learning much faster, while it also facilitates the construction of complex structures (see Fig.1).
Revised: This beneﬁts both the beginning of training (Berthouze & Lungarella, 2004), as the student begins learning much faster, while it also facilitates the construction of complex structures (see Figure 1).

Original: We believe ACCEL provides the best of both worlds: a principled curation method coming withtheoretical guarantees, alongside a mechanism to arbitrarily increase complexity beyond the conﬁnes of a ﬁxed budget or parameterization.
Revised: We believe ACCEL provides the best of both worlds: an evolutionary approach that can generate increasingly complex environments, combined with a regret-based curator which provides theoretical robustness guarantees in equilibrium.

Original: We evaluate ACCEL on a series of challenging procedurally generated grid world environments, where ACCEL demonstrates the ability to rapidly increase complexity while maintaining performance.
Revised: We evaluate ACCEL on a series of challenging procedurally generated grid world environments, where ACCEL demonstrates the ability to rapidly increase complexity while maintaining performance.

Original: Finally, we show that ACCEL makes it possible to train agents capable of transfer to mazes an order of magnitude larger than the training distribution, achieving over double the success rate of the next best baseline
Revised: Finally, we show that ACCEL makes it possible to train agents capable of transfer to mazes an order of magnitude larger than the training distribution, achieving over double the success rate of the next best baseline.

Original:  B ACKGROUND 2. F ROM MDP S TO U NDERSPECIFIED POMDP S
Revised:  B ACKGROUND 2. F ROM MDP S TO U NDERSPECIFIED POMDP S

Original: A Markov Decision Process ( MDP ) is deﬁned as a tuple (cid:104) S, A, T , R , γ (cid:105)
Revised: A Markov Decision Process ( MDP ) is deﬁned as a tuple (cid:104) S, A, T , R , γ (cid:105)

Original: where S and A stand for the sets of states and actions respectively and T : S × A → ∆ ( S ) is a transition function representing the probability that the system/agent transitions 
Revised: where S and A stand for the sets of states and actions respectively and T : S × A → ∆ ( S ) is a transition function representing the probability that the system/agent transitions from a state s t ∈ S to s t +1 ∈ S given action a t ∈ A .

Original: Each transition also induces an associated reward r t generated by a reward function R : S → R , and γ is a discount factor.
Revised: Each transition also induces an associated reward r t generated by a reward function R : S → R , and γ is a discount factor.

Original: When provided with an MDP, the goal of Reinforcement Learning (RL, Sutton & Barto, 1998) is to learn a policy π that maximizes expected discounted reward, i.e. E [ (cid:80)
Revised: When provided with an MDP, the goal of Reinforcement Learning (RL, Sutton & Barto, 1998) is to learn a policy π that maximizes expected discounted reward, i.e. E [ (cid:80)

Original: Ti =0 r t γ t ] .
Revised: Ti =0 r t γ t ] .

Original: Despite the generality of the MDP framework, it is often an unrealistic model for real world environments.
Revised: Despite the generality of the MDP framework, it is often an unrealistic model for real world environments.

Original: First, it assumes full observability of the state, which is often impossible in practice.
Revised: First, it assumes full observability of the state, which is often impossible in practice.

Original: This is addressed in partially observable MDPs, or POMDPs, which include an observation function
Revised: This is addressed in partially observable MDPs, or POMDPs, which include an observation function

Original: I : S
Revised: I : S

Original: → O which maps the true state (which is unknown to the agent) to a (potentially noisy) set of observations O .
Revised: → O which maps the true state (which is unknown to the agent) to a (potentially noisy) set of observations O .

Original: Secondly, the traditional MDP framework assumes a single reward and transition function, which are ﬁxed throughout learning.
Revised: Secondly, the traditional MDP framework assumes a single reward and transition function, which are ﬁxed throughout learning.

Original: Instead, in the real world, agents may experience variations not seen during training, which makes it crucial that policies are capable of robust transfer.
Revised: Instead, in the real world, agents may experience variations not seen during training, which makes it crucial that policies are capable of robust transfer.

Original: To address both of these issues, we use the recently introduced Underspeciﬁed POMDP, or UPOMDP, given by M = (cid:104) A, O, Θ , S M , T M , I M , R M , γ (cid:105) .
Revised: To address both of these issues, we use the recently introduced Underspeciﬁed POMDP, or UPOMDP, given by M = (cid:104) A, O, Θ , S M , T M , I M , R M , γ (cid:105) .

Original: This deﬁnition is identical to a POMDP with the addition of Θ to represent the free-parameters of the environment.
Revised: This deﬁnition is identical to a POMDP with the addition of Θ to represent the free parameters of the environment.

Original: These parameters can be distinct at every time step and incorporated into the transition function T M : S ×
Revised: These parameters can be distinct at every time step and incorporated into the transition function T M : S ×

Original: A × Θ → ∆ ( S ) .
Revised: A × Θ → ∆ ( S ) .

Original: Following
Revised: Following

Original: Jiang et al.
Revised: Jiang et al.

Original: (2021a)
Revised: (2021a)

Original: we deﬁne a level M θ as an environment resulting from a ﬁxed θ .
Revised: we deﬁne a level M θ as an environment resulting from a ﬁxed θ .

Original: We deﬁne the value of π in M θ to be V θ
Revised: We deﬁne the value of π in M θ to be V θ

Original: ( π ) =
Revised: ( π ) =

Original: E [ (cid:80) Ti =0 r t γ t ] where r t are the rewards achieved by π in M θ .
Revised: E [ (cid:80) Ti =0 r t γ t ] where r t are the rewards achieved by π in M θ .

Original: UPOMDPs beneﬁt from their generality, since Θ can represent possible dynamics (for example in sim2real (Peng et al., 2017; OpenAI et al., 2019; Andrychowicz et al., 2020)), changes in observations, different reward functions or differing game maps in procedurally generated environments.
Revised: UPOMDPs beneﬁt from their generality, since Θ can represent possible dynamics (for example in sim2real (Peng et al., 2017; OpenAI et al., 2019; Andrychowicz et al., 2020)), changes in observations, different reward functions or differing game maps in procedurally generated environments.

Original: 2. M ETHODS FOR U NSUPERVISED E NVIRONMENT D ESIGN
Revised: 2. M ETHODS FOR U NSUPERVISED E NVIRONMENT D ESIGN

Original: The goal of Unsupervised Environment Design (UED, Dennis et al., 2020) is to generate a series of levels that form a curriculum for a student agent,
Revised: The goal of Unsupervised Environment Design (UED, Dennis et al., 2020) is to generate a series of levels that form a curriculum for a student agent,

Original: such that the student agent is capable of transfer, by maximizing some utility function U t ( π, θ ) .
Revised: such that the student agent is capable of transfer, by maximizing some utility function U t ( π, θ ) .

Original: In the case of domain randomization, the utility function is simply:
Revised: In the case of DR, the utility function is simply:

Original: [Equation]
Revised: [Equation]

Original: for any constant C .
Revised: for any constant C .

Original: When learning a teacher, recent approaches proposed to use objectives seeking to maximize regret , deﬁned as the difference between the expected return of the current policy and the optimal policy, ie:
Revised: When learning a teacher, recent approaches proposed to use objectives seeking to maximize regret , deﬁned as the difference between the expected return of the current policy and the optimal policy, ie:

Original: [Equation]
Revised: [Equation]

Original: Unlike other objectives, which may promote unsolvable environments, regret-based objectives have been shown to promote the simplest possible environments that the agent cannot currently solve (Dennis et al., 2020).
Revised: Unlike other objectives, which may promote unsolvable environments, regret-based objectives have been shown to promote the simplest possible environments that the agent cannot currently solve (Dennis et al., 2020) in a range of settings.

Original: However, since we do not have access to π ∗ , a key challenge in UED algorithms utilizing objectives inspired by Equation 2 is to approximate the regret.
Revised: However, since we do not have access to π ∗ , a key challenge in UED algorithms utilizing objectives inspired by Equation 2 is to approximate the regret.

Original: Recently, the Prioritized Level Replay (PLR, Jiang et al. , 2021b;a) algorithm introduced an additional teacher agent in the form of a curator , forming a “dual curriculum game”.
Revised: Recently, the Prioritized Level Replay (PLR, Jiang et al., 2021b;a) algorithm introduced an additional teacher agent in the form of a curator , forming a “dual curriculum game”.

Original: The curator maintains a buffer of previously experienced levels andactively selects levels to be replayed by the student policy using objectives approximating regret.
Revised: The curator maintains a buffer of previously experienced levels and selects levels to be replayed by the student policy using objectives approximating regret.

Original: One of the objectives used by PLR is Positive Value Loss , given by:
Revised: One of the objectives used by PLR is Positive Value Loss , given by:

Original: [Equation]
Revised: [Equation]

Original: where λ and γ are the GAE (Schulman et al., 2016) and MDP discount factors respectively, and δ t , the TD-error at timestep t .
Revised: where λ and γ are the Generalized Advantage Estimation (GAE, Schulman et al. and MDP discount factors respectively, and δ t , the TD-error at timestep t .

Original: Since Positive Value Loss approximates regret, if the student trains solely on curated levels (i.e. does not take gradient steps on levels from the generator), then PLR achieves robustness guarantees.
Revised: Since Positive Value Loss approximates regret, if the student trains solely on curated levels (i.e. does not take gradient steps on levels from the generator), then PLR achieves robustness guarantees in equilibrium.

Original: More formally, if S t =
Revised: More formally, if S t =

Original: Π is the strategy set of the student and S t =
Revised: Π is the strategy set of the student and S t =

Original: Θ is the strategy set of the teacher (in this case the curator), then (by Corollary 1 of Jiang et al.
Revised: Θ is the strategy set of the teacher (in this case the curator), then (by Corollary 1 of Jiang et al.

Original: (2021a)), in equilibrium the resulting student policy  converges to a minimax regret policy, ie:
Revised: (2021a)) , in equilibrium the resulting student policy π converges to a minimax regret policy, ie:

Original: [Equation]
Revised: [Equation]

Original:  PLR not only leads to robustness in equilibria but also empirically produces policies with strong generalization capabilities.
Revised: Empirically PLR has also been shown produce policies with strong generalization capabilities 1 , yet it’s main weakness is that it still relies on randomly sampling useful levels.

Original: Next, we introduce our new approach which seeks to leverage the curator to produce batches of high regret levels.
Revised: Next, we introduce our new approach which seeks to leverage the curator to produce batches of high regret levels.

Original:  C OMPOUNDING C OMPLEXITY BY E DITING L EVELS
Revised:  C OMPOUNDING C OMPLEXITY BY E DITING L EVELS

Original: In this section we propose a new method that helps produce more effective curricula in complex design spaces, inspired by a variety of sources, including regret-based UED methods and evolution.
Revised: In this section we introduce our new method for UED, building on regret-based methods such as PLR. As the dimensionality of the design space increases, it becomes increasingly challenging to randomly sample effective levels for learning—a problem we call “the curse of dimensionality in UED”. Thus, rather than solely rely on curating random levels, we instead look to evolution to produce new batches of levels, by making edits to previously curated ones.

Original: Many previous works have emphasized the importance of training on levels at the “frontier” of difﬁculty (Wang et al., 2019; Jiang et al., 2021b;
Revised: This is a direct attempt to produce more levels at the “frontier” of agent capabilities, which has been shown to be useful in a variety of recent works (Wang et al., 2019; Jiang et al., 2021b;

Original: Zhang et al., 2020).
Revised: Zhang et al., 2020).

Original: This is often implemented via heuristics, for  example, POET pre-ﬁlters levels using handcrafted criteria to have a reward in the range [50 , 300] .Instead, we seek a more general approach and follow the Value Correction
Revised: Evolutionary methods are suitable for this, yet often require heuristics to deﬁne a workable ﬁtness function. For example, POET pre-ﬁlters levels using handcrafted criteria to have a reward in the range [50 , 300] .

Original: Hypothesis from Jiang et al. (2021b), using  the  Positive Value Loss to assess learning potential.
Revised: We propose a more general approach using regret in the form of Positive Value Loss to assess learning potential.

Original: To see the impact of PLR on a simple example, we include a visualization in 13 in the Appendix. Indeed, it is critically important that the agent experiences frontier levels to learn efﬁciently, which may be a challenge as the dimensionality of the design space increases—a problem we call “the curse of dimensionality in UED”. However, if it is possible to edit levels, then it may be possible to produce an entire batch of useful levels from a single example.
Revised: We call our method Adversarially Compounding Complexity by Editing Levels , or ACCEL. Algorithm

Original: Input: Buffer size K , initial ﬁll ratio ρ .
Revised: Input: Buffer size K , initial ﬁll ratio ρ .

Original: Initialize: Initialize policy π ( φ ) , level buffer, Λ .
Revised: Initialize: Initialize policy π ( φ ) , level buffer Λ .

Original: # Initial Data Collection
Revised: # Initial Data Collection

Original: Sample K ∗ ρ initial levels.
Revised: Sample K ∗ ρ initial levels.

Original: # Main Training Loop while not converged do Sample replay decision d ∼ P D ( d ) if d = 0 then Sample level θ from level generator
Revised: # Main Training Loop while not converged do Sample replay decision d ∼ P D ( d ) if d = 0 then Sample level θ from level generator

Original: Collect π ’s trajectory τ on θ , with a stopgradient φ ⊥ Compute PLR score, SP = score ( τ, π )
Revised: Collect π ’s trajectory τ on θ , with a stopgradient φ ⊥ Compute PLR score, S = score ( τ, π )

Original: Update Λ with θ using score S P else Sample a replay level, θ ∼ Λ
Revised: Add θ to Λ if score S meets threshold else Sample a replay level, θ ∼ Λ

Original: Collect policy trajectory τ on θ
Revised: Collect policy trajectory τ on θ

Original: Update π with rewards R ( τ )
Revised: Update π with rewards R ( τ )

Original: Edit Levels, to produce θ (cid:48)
Revised: Edit θ to produce θ (cid:48)

Original: Collect π ’s trajectory τ on θ (cid:48) , with a stopgradient φ ⊥ Compute PLR score, SP = score ( τ, π )
Revised: Collect π ’s trajectory τ on θ (cid:48) , with a stopgradient φ ⊥ Compute PLR score, S = score ( τ, π )

Original: Update Λ with θ (cid:48) using score S P (Optionally) Update Editor using score SP end end
Revised: Add θ (cid:48) to Λ if score S meets threshold (Optionally) Update Editor using score S end end

Original: We now introduce our new algorithm, which we call Adversarially Compounding Complexity by

Original: Editing Levels , or ACCEL.

Original: The key idea of ACCEL is to introduce an editor , which produces new levels for the agent by making edits to levels  sampled by the curator.
Revised: The key idea of ACCEL is to introduce an editor , which produces new levels for the agent by making edits to levels previously sampled by the curator.

Original: Indeed, ACCEL can be seen an evolutionary process, whereby the “ﬁtness” function is (approximate) regret, since levels only stay in the “population” (or level replay buffer) if they meet the criteria for curation. Evolution has led to many successes in other domains (Stanley et al., 2019;
Revised: Editing involves making a handful of changes (e.g. adding/removing tiles on a maze), but could be extended to generative models (e.g. perturbations in a latent space).

Original: Pugh et al., 2016), and even proven useful for UED with the POET algorithm. In contrast to POET, ACCEL only trains a single agent , compared to a population of twenty. This has the beneﬁt of having 20 × lower computational cost while also producing a single generally capable agent. In addition, POET uses an approximate minimax adversarial objective (as opposed to minimax regret ) which produces unsolvable environments (Dennis et al., 2020), and thus requires heuristics to succeed in the original paper.
Revised: Equipped with the ability to edit levels, it is possible to produce an entire batch of useful levels from a single example, while incrementally increasing complexity.

Original: We consider both a learned editor, optimizing for Positive Value Loss (Equation 3), and a random one.
Revised: We consider both a learned editor, optimizing for Positive Value Loss (Equation 3), and a random one.

Original: Like with robust PLR from Jiang et al. (2021a)
Revised: Following Robust PLR (Jiang et al., 2021a)

Original: we do not take gradient steps onthe edited level.
Revised: we do not initially train on edited levels.

Original: Instead, we evaluate them and  add them to the replay bufferfor curation if they meet the threshold for the scoring function.
Revised: Instead, we evaluate them and only add them to the replay buffer if they meet the threshold for the scoring function (high regret).

Original: Finally, we consider two  criteria for editing levels, both applied to the most recent replaybatch: “easy” where we edit the levels which have the highest return and lowest approximate regret, essentially those which the agent can  solve with low future learning potential,  and “batch” where we use the entire batch.
Revised: We consider two different criteria for selecting which replay levels to edit: those which the agent can now solve with low future learning potential, approximated as return minus regret, which we call “easy”, and “batch” where we use the entire batch.

Original: The full procedure is shown in Algorithm 1.
Revised: The full procedure is shown in Algorithm 1.

Original: Though backtrack if it goes left.

Original: We posit that editing is effective for two reasons.
Revised: We posit that editing is effective for two reasons.

Original: First,in some cases, a small incremental change to an environment can lead to dramatic changes in their characteristics (Sturtevant et al., 2020), which may move those that are currently too hard or too easy towards the frontier of the agent’s capabilities.
Revised: First, small incremental changes to a level can lead to a diverse batch of new ones (Sturtevant et al., 2020), which may move those that are currently too hard or too easy towards the frontier of the agent’s capabilities.

Original: This may also prevent overﬁtting, for example in Fig.3 we see three levels generated by ACCEL in the MiniGrid environment.
Revised: This may also prevent overﬁtting, for example, in Figure 3 we see three levels generated by ACCEL in a grid world environment (Chevalier-Boisvert et al., 2018).

Original: Each is an edit of the same level and  requires the agent to explore in a different fashion.
Revised: Each is an edit of the same level, and has a similar initial observation, yet requires the agent to explore in a different fashion to reach the goal.

Original: Training on these environments simultaneously will teach the agent to actively explore the environment, backtracking where necessary to ﬁnd the goal.
Revised: Training on these environments simultaneously will teach the agent to actively explore the environment.

Original: Second, making edits outside of the direct trajectory of the agent can be seen as a form of data augmentation, which has been shown to improve sample efﬁciency in RL (Laskin et al., 2020; Kostrikov et al., 2021; Raileanu et al., 2020).
Revised: Second, making edits outside of the direct trajectory of the agent can be seen as a form of data augmentation, which has been shown to improve sample efﬁciency in RL (Laskin et al., 2020; Kostrikov et al., 2021; Raileanu et al., 2020), since it changes the observation but not the optimal policy.

Original: A key beneﬁt in comparison to prior work is that since we are building upon the robust version of
Revised: ACCEL is an Evolutionary algorithm, whereby the “ﬁtness” is (approximate) regret, since levels only stay in the “population” (or level replay buffer) if they meet the criteria for curation. Evolution has led to many successes in other domains (Stanley et al., 2019;

Original: PLR (Jiang et al., 2021a), our approach maintains the same theoretical guarantees:
Revised: Pugh et al., 2016), and even proven useful for UED with the POET algorithm.

Original: Remark 1.
Revised: Compared to POET we have two key differences: ﬁrst, we have a population of levels but not a population of agents, thus we have a single, generally capable agent.

Original: If the procedure described in Algorithm 1 ﬁnds a Nash equilibrium, then the student policy is following a minimax regret strategy. Note that this is a somewhat trivial remark: since we do not train on levels produced by the editor, the procedure resembles PLR ⊥ from Jiang et al.
Revised: This likely leads to a lower computational cost. In addition, ACCEL uses a minimax regret (rather than minimax) objective, which means we do not need handcrafted rules to select levels, since optimizing for regret naturally promotes levels at the frontier of agent’s capability.

Original: (2021a), thus inherits the  guarantees  from Corrollary 3.3 (in Jiang et al.
Revised: Indeed, training on high regret levels also means that ACCEL inherits the robustness guarantees in equilbrium from Robust PLR (Corrollary 1 in Jiang et al.

Original: (2021a)).
Revised: (2021a)):

Revised: Remark 1.

Revised: If the procedure described in Algorithm 1 ﬁnds a Nash equilibrium, then the student policy is following a minimax regret strategy.

Original: Nonetheless, this is in stark contrast with other evolutionary approaches, which rely solely on empirical results.
Revised: This is in stark contrast with other evolutionary approaches, which rely solely on empirical results.

Original: We are now ready to test ACCEL in practice.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: In our experiments we seek to answer the following two questions: 1) Can ACCEL lead to sample efﬁcient learning in complex design spaces?
Revised: In our experiments we seek to answer the following two questions: 1) Can ACCEL lead to sample efﬁcient learning in complex design spaces?

Original: 2) Can ACCEL compound complexity to asymptotically produce agents capable of solving challenging human-designed environments?
Revised: 2) Can ACCEL compound complexity to asymptotically produce agents capable of zero-shot transfer to challenging out-of-distribution environments?

Original: We conduct a series of experiments in grid world environments, as have been used in previous UED works (Dennis et al., 2020; Jiang et al., 2021a).
Revised: We conduct a series of experiments in grid world environments, as have been used in previous UED works (Dennis et al., 2020; Jiang et al., 2021a).

Original: These environments are made challenging by high dimensional observations and sparse rewards, thus they are often used to test state-of-the-art exploration methods (Raileanu & Rockt ¨ aschel, 2020; Zhang et al., 2021b; Flet-Berliac et al., 2021).
Revised: These environments are made challenging by high dimensional observations and sparse rewards, thus they are often used to test state-of-the-art exploration methods (Raileanu & Rockt¨aschel, 2020; Zhang et al., 2021b; Flet-Berliac et al., 2021).

Original: In all cases, we seek to train a student agent via Proximal Policy Optimization (PPO, Schulman et al., 2016-2018), with a ResNet policy (He et al., 2016) as originally proposed in Espeholt et al.
Revised: In all cases, we seek to train a student agent via Proximal Policy Optimization (PPO, Schulman et al., 2016-2018), with a ResNet policy (He et al., 2016) as originally proposed in Espeholt et al.

Original: To evaluate the quality of the curricula, we show all performance with respect to the number of student gradient updates as opposed to total environment interactions, which is often comparable for PLR and ACCEL.
Revised: To evaluate the quality of the curricula, we show all performance with respect to the number of student gradient updates as opposed to total environment interactions, which is often comparable for PLR and ACCEL.

Original: For a full list of hyperparameters for each experiment please see Table 6 in
Revised: For a full list of hyperparameters for each experiment please see Table 6 in

Original: Section B.3.
Revised: Section B.3.

Original: As baselines we consider the following:
Revised: As baselines we consider the following:

Original: • Domain Randomization (DR) :
Revised: • Domain Randomization (DR) :

Original: Randomly sampling from a parameterized distribution.
Revised: Randomly sampling from a parameterized distribution.

Original: • Prioritized Level Replay (PLR) :
Revised: • Prioritized Level Replay (PLR) :

Original: We use Robust PLR from Jiang et al.
Revised: We use Robust PLR from Jiang et al.

Original: (2021a).
Revised: 2021a

Original: • PAIRED : Placing a given number of blocks, using the algorithm described in Dennis et al.
Revised: • PAIRED : Placing a ﬁxed number of blocks, using the algorithm described in Dennis et al.

Original: • Minimax Adversarial : Placing blocks using an adversarial objective seeking to minimize return.
Revised: • Minimax Adversarial : Placing blocks using an adversarial objective seeking to minimize return.

Original: As in Dennis et al.
Revised: As in Dennis et al.

Original:  we use a single minimax agent to demonstrate the ﬂaws in the POET objective when it is not combined with hand-coded constraints on the levels generated.
Revised: (2020), we use a single minimax represent the POET objective when it is not combined with hand-coded constraints on the levels generated.

Original: We leave the comparison to population-based methods to future work due to the computational expense.
Revised: We leave the comparison to population-based methods to future work due to the computational expense.

Original: 4. L EARNING WITH L AVA
Revised: 4. L EARNING WITH L AVA

Original: [Figure]
Revised: [Figure]

Revised: [Figure]

Original: We begin with a simple grid environment, whereby the agent must navigate to a goal in the presence of lava blocks.
Revised: We begin with a simple grid environment, whereby the agent must navigate to a goal in the presence of lava blocks.

Original: The grid is small, only 7x7, however, it is challenging for RL agents since exploring with random actions often leads to instant death, which makes it challenging to receive feedback via the sparse reward.
Revised: The grid is small, only 7x7, however, it is challenging for RL agents since exploring with random actions often leads to instant death, which makes it unlikely to receive a signal from the sparse reward.

Original: Indeed, this makes the choice of DR parameterization crucial, since sampling too many blocks early on will be prohibitive for learning.
Revised: Indeed, this makes the choice of DR parameterization crucial, since sampling too many blocks early on will be prohibitive for learning.

Original: We compare two different parameterizations: “Binomial” where the agent samples from { lava, none } for 20 steps, and “Uniform” where the agent samples the number of tiles to place from the range [0,20].
Revised: We compare two different parameterizations: “Binomial” where the agent samples from { lava, none } for 20 steps, and “Uniform” where the agent samples the number of tiles to place from the range [0,20].

Original: For ACCEL, we use a generator that produces empty rooms and then proceed to edit the levels to add (or remove) lava blocks.
Revised: For ACCEL, we use a generator that produces empty rooms and then proceed to edit the levels to add (or remove) lava blocks.

Original: The environment is built with MiniHack, thus the agent has a global observation (details in the Appendix,
Revised: The environment is built with MiniHack, thus the agent has a global observation (details in the Appendix,

Original: Section: B.1).
Revised: Section: B.1).

Original: We ran each method for ﬁve seeds, showing the results in Fig 4.
Revised: We ran each method for ﬁve seeds, showing the results in Fig 4.

Original: As we see, ACCEL quickly produces levels with more lava than the other methods, while also getting near-perfect return on its training distribution.
Revised: As we see, ACCEL quickly produces levels with more lava than the other methods, while also getting near-perfect return on its training distribution.

Original: Interestingly, with the Uniform parameterization, we see that PLR is able to produce a similar training proﬁle to ACCEL, but achieves a lower value in every individual metric.
Revised: Interestingly, with the Uniform parameterization, we see that PLR is able to produce a similar training proﬁle to ACCEL, but achieves a lower value in every individual metric.

Original: The methods with learned generators (PAIRED and minimax) fail to learn anything, thus are unable to form a curriculum (Fig. 4).
Revised: The methods with learned generators (PAIRED and minimax) fail to learn anything, thus are unable to form a curriculum (Figure 4).

Original: Finally on the right we took a snapshot of the level replay buffer for ACCEL, where we clearly see that the levels which have been edited more have higher approximate regret.
Revised: Finally on the right we took a snapshot of the level replay buffer for ACCEL, where we clearly see that the levels which have been edited more have higher approximate regret.

Original: After one thousand PPO updates (around 20M timesteps) we tested each agent on a series of test tasks, which we show in the Appendix (see Section A.2).
Revised: After one thousand PPO updates (around 20M timesteps) we tested each agent on a series of test tasks, which we show in the Appendix (see Section A.3).

Original: We thus have answered our ﬁrst question: in a design space with a high proportion of challenging levels, ACCEL is able to build an effective curriculum which quickly facilitates learning on the full distribution.
Revised: We thus have answered our ﬁrst question: in a design space with a high proportion of challenging levels, ACCEL is able to build an effective curriculum which quickly facilitates learning on the full distribution.

Original: 4. P ARTIALLY O BSERVABLE N AVIGATION
Revised: 4. P ARTIALLY O BSERVABLE N AVIGATION

Original: To answer the second question, we now scale toa larger environment, using the MiniGrid (ChevalierBoisvert et al., 2018) setting originally introduced in Dennis et al.
Revised: To answer the second question, we now scale to the MiniGrid (Chevalier-Boisvert et al., 2018) setting originally introduced in Dennis et al.

Original: Despite being a conceptually simple environment, this is a largescale experiment: our agents train for 20k updates (around 350M steps, see Table5), learning a recurrent policy with a 147 dimension partially observable observation.
Revised: Despite being a conceptually simple environment, this is a large experiment: our agents train for 20k updates (around 350M steps, see Table 5), learning an LSTM-based policy with a 147 dimension partially observable observation.

Original: As such, the maze environment has become a prominent benchmark for UED algorithms (Dennis et al., 2020; Jiang et al., 2021a).

Original: In previous works, DR used the “ﬁxed” parameterization, which Jiang et al. (2021a) showed that DR is particularly sensitive to the number of blocks placed. Instead, we use the Uniform parameterization  which ﬁrst samples the number of blocks to place, ranging from zero (an empty room) to sixty.
Revised: We use the Uniform parameterization for DR which ﬁrst samples the number of blocks to place, ranging from zero (an empty room) to sixty, since previous works showed that DR is sensitive to the number of blocks placed (Jiang et al., 2021a).

Original: We use the same generator for DR, PLR and ACCEL.

Original: We also consider a version of which begins with empty rooms, as in the lava environment.

Original: Forboth versions of ACCEL we  edit the block locations (adding or removing them) as well of the goal location.
Revised: For ACCEL we begin with empty rooms and randomly edit the block locations (adding or removing them) as well of the goal location.

Original: After replay, we edit the “easy” levels, essentially moving levels back to the frontier once their learning potential has been reduced.
Revised: After replay, we edit the “easy” levels, essentially moving levels back to the frontier once their learning potential has been reduced.

Original: [Figure]
Revised: [Figure]

Original: In Fig.5 we show the training performance, where ACCEL does exactly as intended—it compounds initial complexity to quickly trainagents capable of performing well on  the full distribution.
Revised: In Figure 5.a) we show the training performance, where ACCEL does exactly as intended—it compounds initial complexity to ultimately train on levels with high block count and long paths to the goal.

Original: Indeed, ACCEL learns more efﬁciently on the DR distribution than DR and PLR, the agents that explicitly train on it. Meanwhile, ACCEL produces more levels with more blocks and longer paths to the goal. Inevitably, we see that the edited levels take an increasing share of the replay buffer.
Revised: This can be seen in 5.b), where ACCEL produces more structured mazes than the baselines.

Original: After 20k training steps, we evaluate the agents zero-shot on a series of held-out environments as used in prior works (see Fig.12), with theperformance shown in Fig.6. We include bars showing the mean and sem for each envronment, as well as the aggregate performance using the recently introduced rliable
Revised: We evaluate all ﬁve methods zero-shot on a series of held-out environments as used in prior works (see Figure 16), with the mean and sem per environment shown in Figure 6. For DR, PLR and ACCEL the evaluation is after 20k student updates, thus it solely compares the quality of the curriculum, while we use the Minimax and PAIRED results from Jiang et al.

Original: Agarwal et al. (2021b).
Revised: (2021a)

Original:  As we see,both versions of ACCEL performs at least as well as the next best baseline in almost all settings, with notable outperformance on the more complex Labyrinth and Maze environments.
Revised: at 250M training steps ( > 30k updates). As we see, ACCEL performs as least as well as the next best method in almost all settings, with particular strength in the more complex Labyrinth and Maze environments.

Original: This translates to signiﬁcantly stronger performance at the aggregate level, with an IQM near 100% solved rate compared to below 80% for all baselines.

Original: [Figure] 
Revised: [Figure] o l v e d

Original:  a t 
Revised: R a t e

Original: [Figure]
Revised: [Figure]

Original: 16Rooms

Original: [Figure]
Revised: [Figure]

Original: 16Rooms [Figure] [Figure] [Figure]
Revised: To evaluate the performance at the aggregate level, we make use of the recently introduced rliable library (Agarwal et al., 2021b) in Figure 7.b).

Original: SmCorridor [Figure] LgCorridor 1 Labyrinth 1 Labyrinth [Figure]
Revised: At the aggregate level the strength of ACCEL is clear, with an IQM 2 near 100% solved rate compared to below 80% for PLR, with a probability of improvement of 80.2%.

Original: Maze

Original: [Figure] Maze [Figure] [Figure]
Revised: It is clear that ACCEL is signiﬁcantly stronger in these test environments.

Original: Next we consider a signiﬁcantly more challenging environment—we use a larger version of the “PerfectMaze”, a procedurally-generated maze environment, shown in Fig.7.a).
Revised: Next we consider an even more challenging setting—we use a larger version of the “PerfectMaze”, a procedurally-generated maze environment, shown in Figure 7.a).

Original: The maze is a 51x grid, an order of magnitude larger than the training environments [Figure] and we use a maximum episode length of over 5k steps.
Revised: The maze has 51x51 tiles, an order of magnitude larger than the training environment, and has a maximum episode length of over 5k steps.

Original: This represents a daunting navigation challenge, requiring extensive use of memory so as not to get lost in a loop of repeatedly exploring the same paths.
Revised: This is a daunting navigation challenge, requiring extensive use of memory so as not to get lost in a loop of repeatedly exploring the same paths.

Original: We evaluate the agents at the 20k checkpoint, testing each seed for 100 episodes, showing the mean and standard error in Fig.7.a).
Revised: We evaluate the agents at the checkpoint from Figure 6, testing each seed for 100 episodes, showing the mean and standard error in Figure 7.a).

Original: As we see, ACCEL signiﬁcantly outperforms all baselines, achieving52% success rate compared to the next best 25% for PLR.
Revised: Both versions of ACCEL signiﬁcantly outperform all baselines, achieving success rates of 53% and 52% compared to the next best 25% for PLR, while all other methods fail. Notably, ACCEL appears to approximately follow the “left-hand” rule for solving mazes.

Original: In Fig.7.b) we show example levels from each method, where  we see that ACCEL produces more complex mazes with higher block count, regardless of the DR generator used.
Revised: [Figure] What if we edit DR levels? We also consider an ablation of ACCEL where instead of beginning with empty rooms, we instead begin with levels sampled from the DR distribution.

Original: 1 Note that PLR is capable of sampling 60 block levels, but it will infrequently sample those that  also contain a useful (solvable) path to the goal, whereas ACCEL can produce entire batches of challenging levels by editing previously complex ones.
Revised: In Figure 8 we show the performance of ACCEL, DR and PLR on the same generator distribution during training, where we see that ACCEL outperforms both. We also tested this version of ACCEL on the same held out tasks, where it achieved a mean performance comparable to the version presented in Figure 6, while it also achieved 52% success on the large perfect maze.

Original: For additional ablations see the Appendix (see Section A.3).
Revised: We conducted other ablations such as using a learned editor, or editing the full batch, with only small changes in performance (see Section A.4). We believe this shows the strength of ACCEL, that it is robust to a multitude of factors, even the generator distribution.

Original: 4. D ISCUSSION AND L IMITATIONS
Revised: 4. D ISCUSSION AND L IMITATIONS

Original: In our experiments we have demonstrated that ACCEL is capable of forming highly effective curricula in two challenging navigation environments.
Revised: In our experiments we have demonstrated that ACCEL is capable of forming highly effective curricula in two challenging navigation environments.

Original: In the ﬁrst, we showed ACCEL can facilitate learning in challenging design space with a high proportion of hard levels, which could have an impact in improving exploration in safety-critical settings.
Revised: In the ﬁrst, we showed ACCEL can facilitate learning in a design space with a high proportion of hard levels, which could have an impact in improving exploration in safety-critical settings.

Original: In the second, we showed it is possible to produce complex mazes which facilitate generalization to human-designed ones, scaling to mazes an order of magnitude larger than the training levels.
Revised: In the second, we showed it is possible to produce complex mazes which facilitate zero-shot transfer to human-designed ones, scaling to environments an order of magnitude larger than the training environment.

Revised: This is made possible since ACCEL produces a high frequency of solvable mazes with a high block count, regardless of the DR generator used.

Revised: Note that PLR is capable of sampling 60 block levels, but it will infrequently sample those that also contain a useful (solvable) path to the goal.

Original: We thus believe  ACCEL would be an effective method for training agents in more open-ended UED design spaces.
Revised: We thus believe we have shown evidence that ACCEL would be an effective method for training agents in more open-ended UED design spaces.

Original: However, as with any method, ACCEL comes with limitations.
Revised: However, as with any method, ACCEL comes with limitations.

Original: In the lava experiments, our approach includes an inductive bias with the ability to begin with a simple base case (an empty room).
Revised: Our approach includes an inductive bias with the ability to begin with a simple base case (an empty room).

Original: This may not always be possible in practice, while in some cases the simple case (in terms of entities placed in the environment) may actually be a more difﬁcult environment to solve (for example in a Hide and Seek game).
Revised: This may not always be possible in practice, while in some settings the simplest example (in terms of entities placed in the environment) may actually be a more difﬁcult environment to solve (for example in a Hide and Seek game).

Original: In addition, our experiments thus far only consider navigation tasks. The MiniGrid experimentswe include here are one of the largest settings used in UED,but for the ﬁeld more broadly to become useful for real-world problems it will be necessary to test new environments.
Revised: In addition, our experiments thus far only consider navigation tasks, and while the MiniGrid experiments are one of the largest settings used in UED, for the ﬁeld more broadly to become useful for real-world problems it will be necessary to test new environments.

Original: On the algorithmic side, a potential limitation of ACCEL isthat there may be a lack of diversity in the level replay buffer.
Revised: On the algorithmic side, a potential limitation of ACCEL is there may be a lack of diversity in the level replay buffer.

Original: In this work we do not explicitly optimize for diversity, but instead, seek to reduce the impact throughchanging level replay hyperparameters.
Revised: In this work we do not explicitly optimize for diversity, but instead, seek to reduce the impact through level replay hyperparameters (see the Appendix, Figure 21).

Original: For example, we continue to sample DR levels during training and for the lava experiments we also use a larger buffer. We demonstrate the impact of these design choices in the Appendix (see: Fig 15). We thus believe in order to scale
Revised: However, scaling

Original: ACCEL further it will be crucial to include a mechanism to  encourage diversity.
Revised: ACCEL further may require a mechanism to directly encourage diversity.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Our work straddles a variety of interrelated ﬁelds, which we discuss in this section.
Revised: Our work straddles a variety of interrelated ﬁelds, which we discuss in this section.

Original: For a summary of the most closely related methods see Table 1.
Revised: For a summary of the most closely related methods see Table 1.

Original: Our paper focuses on testing agents on distributions of environments, which has long been known to be crucial to evaluate the generalization capability of RL agents (Whiteson et al., 2009).
Revised: Our paper focuses on testing agents on distributions of environments, which has long been known to be crucial to evaluate the generalization capability of RL agents (Whiteson et al., 2009).

Original: The failure of agents in this setting has recently drawn considerable attention (Packer et al., 2019; Igl et al., 2019;
Revised: The failure of agents in this setting has recently drawn considerable attention (Packer et al., 2019; Igl et al., 2019;

Original: Cobbe et al., 2020; Agarwal et al., 2021a; Zhang et al., 2018b; Ghosh et al., 2021), with policies often failing to adapt to changes in the observation (Song et al., 2020), dynamics (Ball et al., 2021) or reward (Zhang et al., 2018a).
Revised: Cobbe et al., 2020; Agarwal et al., 2021a; Zhang et al., 2018b; Ghosh et al., 2021), with policies often failing to adapt to changes in the observation (Song et al., 2020), dynamics (Ball et al., 2021) or reward (Zhang et al., 2018a).

Original: In this work, we seek to provide agents with a set of training levels to produce a policy that is robust to these variations.
Revised: In this work, we seek to provide agents with a set of training levels to produce a policy that is robust to these variations.

Original: In particular, we focus on the Unsupervised Environment Design (UED, Dennis et al., 2020) paradigm, which shifts from designing agents that can generalize from a ﬁxed distribution of environments towards designing the environments themselves.
Revised: In particular, we focus on the Unsupervised Environment Design (UED, Dennis et al., 2020) paradigm, which shifts from designing agents that can generalize from a ﬁxed distribution of environments towards designing the environments themselves.

Original: The most popular method for UED is Domain randomization (DR, Jakobi, 1997; Sadeghi & Levine, 2017) which has been particularly successful in areas such as robotics (Tobin et al., 2017; James et al., 2017; Andrychowicz et al., 2020; OpenAI et al., 2019).
Revised: The most popular method for UED is Domain randomization (DR, Jakobi, 1997; Sadeghi & Levine, 2017) which has been particularly successful in areas such as robotics (Tobin et al., 2017; James et al., 2017; Andrychowicz et al., 2020; OpenAI et al., 2019).

Original: This work directly extends Prioritized Level Replay (PLR, Jiang et al., 2021b;a), a method for curating DR levels such that those with high learning potential can be replayed a student agent.
Revised: This work directly extends Prioritized Level Replay (PLR, Jiang et al., 2021b;a), a method for curating DR levels such that those with high learning potential can be replayed a student agent.

Original: PLR is related to TSCL (Matiisen et al., 2020), self-paced learning (Klink et al., 2019; Eimer et al., 2021) and ALP-GMM Portelas et al.
Revised: PLR is related to TSCL (Matiisen et al., 2020), self-paced learning (Klink et al., 2019; Eimer et al., 2021) and ALP-GMM Portelas et al.

Original: (2019), which seek to maintain and update distributions over environment parameterizations.
Revised: (2019), which seek to maintain and update distributions over environment parameterizations.

Original: Very recently it was shown that with a smooth task space, a method similar to PLR is capable of producing generally capable agents in a simulated game world (Team et al., 2021), using large scale compute and Population-Based Training (Jaderberg et al., 2017).
Revised: Very recently it was shown that with a smooth task space, a method similar to PLR is capable of producing generally capable agents in a simulated game world (Team et al., 2021), using large scale compute and Population Based Training (Jaderberg et al., 2017).

Original: However, this work relied on a highly optimized task design space, which is rarely present in practice.
Revised: However, this work relied on a highly optimized task design space, which is rarely present in practice.

Original: Dennis et al.
Revised: Dennis et al.

Original: (2020) introduced the PAIRED algorithm, an elegant approach for UED, whereby an environment adversary optimizes for minimax regret (Savage, 1951), deﬁned as the difference in performance between an antagonist agent (colluding with the adversary) and the protagonist.
Revised: (2020) introduced the PAIRED algorithm, an elegant approach for UED, whereby an environment adversary optimizes for minimax regret (Savage, 1951), deﬁned as the difference in performance between an antagonist agent (colluding with the adversary) and the protagonist.

Original: This guarantees the adversary produces solvable mazes, which allows the protagonist to transfer to unseen environments and even learn to navigate the web (Gur et al., 2021).
Revised: This guarantees the adversary produces solvable mazes, which allows the protagonist to transfer to unseen environments and even learn to navigate the web (Gur et al., 2021).

Original: Adversarial objectives have also been considered in robotics (Pinto et al., 2017).
Revised: Adversarial objectives have also been considered in robotics (Pinto et al., 2017).

Original: POET (Wang et al., 2019; 2020) considers evolving a population of environments, each paired with an agent, using an objective similar to minimax adversarial, which needs to be combined with domain-speciﬁc rules to prevent unsolvable environments from being proposed.
Revised: POET (Wang et al., 2019; 2020) considers evolving a population of environments, each paired with an agent, using an objective similar to minimax adversarial, which needs to be combined with domain-speciﬁc rules to prevent unsolvable environments from being proposed.

Original: We take inspiration from the evolutionary nature of POET but train a single agent , which is beneﬁcial as it takes signiﬁcantly fewer resources, while also removing the agent selection problem.
Revised: We take inspiration from the evolutionary nature of POET but train a single agent , which is beneﬁcial as it takes signiﬁcantly fewer resources, while also removing the agent selection problem.

Original: UED is inherently related to the ﬁeld of Automatic Curriculum Learning (ACL, Portelas et al., 2020;
Revised: UED is inherently related to the ﬁeld of Automatic Curriculum Learning (ACL, Portelas et al., 2020;

Original: Florensa et al., 2017; Baranes & Oudeyer, 2009), which seeks to provide a curriculum of increasingly challenging tasks or goals given a (typically) ﬁxed environment.
Revised: Florensa et al., 2017; Baranes & Oudeyer, 2009), which seeks to provide a curriculum of increasingly challenging tasks or goals given a (typically) ﬁxed environment.

Original: A canonical approach is Hindsight
Revised: A canonical approach is Hindsight

Original: Experience Replay (Andrychowicz et al., 2017) which was shown to be effective for sparse reward tasks.
Revised: Experience Replay (Andrychowicz et al., 2017) which was shown to be effective for sparse reward tasks.

Original: Asymmetric Self Play (Sukhbaatar et al., 2018) takes the form of one agent proposing goals for another, which was shown to be effective for challenging robotic manipulation tasks (OpenAI et al., 2021).
Revised: Asymmetric Self Play (Sukhbaatar et al., 2018) takes the form of one agent proposing goals for another, which was shown to be effective for challenging robotic manipulation tasks (OpenAI et al., 2021).

Original: AMIGo (Campero et al., 2021) and APT-Gen (Fang et al., 2021) provide solutions to problems where the target task is known, providing a curriculum of increasing difﬁculty.
Revised: AMIGo (Campero et al., 2021) and APT-Gen (Fang et al., 2021) provide solutions to problems where the target task is known, providing a curriculum of increasing difﬁculty.

Original: Indeed, many ACL methods emphasize learning to reach goals or states with high uncertainty (Racaniere et al., 2020; Pong et al., 2020; Zhang et al., 2020), either using generative (Florensa et al., 2018) or
Revised: Indeed, many ACL methods emphasize learning to reach goals or states with high uncertainty (Racaniere et al., 2020; Pong et al., 2020; Zhang et al., 2020), either using generative (Florensa et al., 2018) or

Original: [Table]
Revised: [Table]

Original: world models (Mendonca et al., 2021).
Revised: world models (Mendonca et al., 2021).

Original: Unlike these methods, UED approaches seek to fully specify environments, rather than just goals within a ﬁxed environment.
Revised: Unlike these methods, UED approaches seek to fully specify environments, rather than just goals within a ﬁxed environment.

Original: In the symbolic AI community, environment design has been considered as a means to alter an environment to inﬂuence an agent’s decisions (Zhang & Parkes, 2008; Zhang et al., 2009).
Revised: In the symbolic AI community, environment design has been considered as a means to alter an environment to inﬂuence an agent’s decisions (Zhang & Parkes, 2008; Zhang et al., 2009).

Original: This was extended with automated design (Keren et al., 2017; 2019), which seeks to redesign environments given the limitations of agents, to improve their performance.
Revised: This was extended with automated design (Keren et al., 2017; 2019), which seeks to redesign environments given the limitations of agents, to improve their performance.

Original: Unlike these works, ACCEL seeks to automatically design environments in order to produce a curriculum for a learned agent.
Revised: Unlike these works, ACCEL seeks to automatically design environments in order to produce a curriculum for a learned agent.

Original: Our work also closely relates to the ﬁeld of Procedural Content Generation (PCG, Risi & Togelius, 2020; Justesen et al., 2018), where levels are sampled from a distribution.
Revised: Our work also closely relates to the ﬁeld of Procedural Content Generation (PCG, Risi & Togelius, 2020; Justesen et al., 2018), where levels are sampled from a distribution.

Original: Popular PCG settings include the Procgen Benchmark (Cobbe et al., 2020), MiniGrid (Chevalier-Boisvert et al., 2018),
Revised: Popular PCG settings include the Procgen Benchmark (Cobbe et al., 2020), MiniGrid (Chevalier-Boisvert et al., 2018),

Original: Obstacle Tower (Juliani et al., 2019), GVGAI (Perez-Liebana et al., 2019) and the NetHack Learning
Revised: Obstacle Tower (Juliani et al., 2019), GVGAI (Perez-Liebana et al., 2019) and the NetHack Learning

Original: Environment (K¨uttler et al., 2020).
Revised: Environment (K¨uttler et al., 2020).

Original: This work uses the recently proposed MiniHack environment (Samvelyan et al., 2021), which provides a ﬂexible framework for creating diverse environments.
Revised: This work uses the recently proposed MiniHack environment (Samvelyan et al., 2021), which provides a ﬂexible framework for creating diverse environments.

Original: Within the PCG community, automatically generating game levels has been of interest for more than a decade (Togelius & Schmidhuber, 2008).
Revised: Within the PCG community, automatically generating game levels has been of interest for more than a decade (Togelius & Schmidhuber, 2008).

Original: More recently, machine learning has proven to be effective (Summerville et al., 2018;
Revised: More recently, machine learning has proven to be effective (Summerville et al., 2018;

Original: Bhaumik et al., 2020; Liu et al., 2021).
Revised: Bhaumik et al., 2020; Liu et al., 2021).

Original: Related to our work,
Revised: Related to our work,

Original: PCGRL (Khalifa et al., 2020; Earle et al., 2021a) framed level design as an RL problem, designing environments by making incremental changes.
Revised: PCGRL (Khalifa et al., 2020; Earle et al., 2021a) framed level design as an RL problem, designing environments by making incremental changes.

Original: However, it makes use of hand-designed dense rewards, and focuses on the design of levels for human players.
Revised: However, it makes use of hand-designed dense rewards, and focuses on the design of levels for human players.

Original: By contrast, ACCEL seeks to train a student agent and does not require domain-speciﬁc feedback.
Revised: By contrast, ACCEL seeks to train a student agent and does not require domain-speciﬁc feedback.

Original:  C ONCLUSION AND F UTURE W ORK
Revised:  C ONCLUSION AND F UTURE W ORK

Original: In this paper we proposed a new method for Unsupervised Environment Design (UED), ACCEL, which evolves a curriculum by editing previously curated levels.
Revised: In this paper we proposed a new method for Unsupervised Environment Design (UED), ACCEL, which evolves a curriculum by editing previously curated levels.

Original: This makes it possible to constantly generate a wide variety of environments at the frontier of the agent’s capabilities, producing curricula that start simple and quickly compound.
Revised: This makes it possible to constantly generate a wide variety of environments at the frontier of the agent’s capabilities, producing curricula that start simple and quickly compound.

Original: We believe ACCEL offers the best of both worlds: a principled regret-based curriculum that does not require domain-speciﬁc heuristics, alongside an evolutionary process that produces a broad spectrum of complexity catering to the agent’s current capabilities.
Revised: We believe ACCEL offers the best of both worlds: a principled regret-based curriculum that does not require domain-speciﬁc heuristics, alongside an evolutionary process that produces a broad spectrum of complexity catering to the agent’s current capabilities.

Original: In our experiments we showed that ACCEL is capable of efﬁciently training agents that can transfer to a series of human-designed environments, outperforming competitive baselines.
Revised: In our experiments we showed that ACCEL is capable of efﬁciently training agents that can transfer to a series of human-designed environments, outperforming competitive baselines.

Original: For future work, it may be possible to use ideas from Neural Cellular Automata to enhance the editing process (Earle et al., 2021b), possibly making use of controllable editors that can boost speciﬁc properties of levels (Earle et al., 2021a).
Revised: For future work, it may be possible to use ideas from Neural Cellular Automata to enhance the editing process (Earle et al., 2021b), possibly making use of controllable editors that can boost speciﬁc properties of levels (Earle et al., 2021a).

Original: We could also consider generative modelling approaches to level editing, training on PLR levels to produce new frontier levels.
Revised: We could also consider generative modelling approaches to level editing, training on PLR levels to produce new frontier levels.

Original: We did not explore methods to encourage levels to be diverse, but this would likely be important for larger-scale experiments.
Revised: We did not explore methods to encourage levels to be diverse, but this would likely be important for larger-scale experiments.

Original: Another possibility is to actively seek levels which have high “Evolvability” (Gajewski et al., 2019).
Revised: Another possibility is to actively seek levels which have high “Evolvability” (Gajewski et al., 2019).

Original: This could be increased by introducing so-called “extinction events” (Raup, 1986), which have been shown to increase Evolvability (Lehman & Miikkulainen, 2015) and are believed to play a crucial role in natural evolution.
Revised: This could be increased by introducing so-called “extinction events” (Raup, 1986), which have been shown to increase Evolvability (Lehman & Miikkulainen, 2015) and are believed to play a crucial role in natural evolution.

Original: Equipped with some of these ideas, we could go signiﬁcantly further towards open-endedness (Earle et al.).
Revised: Equipped with some of these ideas, we could go signiﬁcantly further towards open-endedness (Earle et al., 2021c).

Original: We believe it may be possible to increase the search space such that MiniHack can make broader use of the richness of the NetHack world, possibly aiding us in making progress in the full game of Nethack—a grand challenge in reinforcement learning. 
Revised: We believe it may be possible to increase the search space such that MiniHack can make broader use of the richness of the NetHack world, possibly aiding us in making progress in the full game of Nethack—a grand challenge in reinforcement learning. S TATEMENT

Revised: We do not perceive any potential ethical issues to arise from this work, beyond those typically associated with reinforcement learning.

Revised: However, a key issue in RL is reproducibility, and as such we will be open sourcing our code alongside the camera ready version of our paper.

Revised: Both environments we used are open sourced (MiniGrid and MiniHack) and we based our code on an open source repo

Original: A A DDITIONAL E XPERIMENTAL R ESULTS A. L EVEL E VOLUTION
Revised: A A DDITIONAL E XPERIMENTAL R ESULTS A. L EVEL E VOLUTION

Original: In Fig 8 and 

Revised: In Fig 9 and

Original: and we show additional levels produced by ACCEL for the MiniHack lava environment and MiniGrid mazes respectively.
Revised: and we show additional levels produced by ACCEL for the MiniHack lava environment and MiniGrid mazes respectively.

Original: Note that in all cases, each incremental step along the evolutionary process produces a level that has high learning potential at that point in time
Revised: Note that in all cases, each incremental step along the evolutionary process produces a level that has high learning potential at that point in time

Original: A. F ULL E XPERIMENTAL R ESULTS
Revised: A. T HE E XPANDING F RONTIER

Revised: Here we analyze the performance of agents on levels produced by ACCEL.

Revised: We have four agent checkpoints, from 5k, 10k, 15k and 20k student gradient updates.

Revised: In Figure 12 we show four generations of a level, where we see that the later generations become harder for the 5k checkpoint, while the 20k checkpoint gets the highest learning signal (Positive Value Loss) from Generation

Revised: [Figure]

Revised: [Figure]

Revised: In Figure 13 we show all generations for the level included in Figure 12, grouped by generation.

Revised: We then show the mean Return and Positive Value loss (PVL) for all four agent checkpoints.

Original: Lava Grid Extended Results In Table 2 and Fig.
Revised: It is clear to see that the later generations have the highest learning potential for the 20k checkpoint, with the lowest return for the 5k checkpoint. Next in Figure 14 we show data for all generations of 20 levels, chosen as those which were present in the 20k checkpoint replay buffer but also had ancestors in the 5k checkpoint buffer. For each checkpoint we plot all levels along the dimensions of Number of Blocks and Shortest Path, with the color corresponding to the solved rate. On the left, the 5k checkpoint agent can only repeatedly solve the shorter path levels with low block count.

Original: 11 we include the full results for the MiniHack lava experiments.
Revised: Later on, the 20k checkpoint agent performs well across the entire space. [Figure] A. F ULL E XPERIMENTAL R ESULTS Lava Grid Extended Results In Table 2 and Figure 15 we include the full results for the MiniHack lava experiments.

Original: The ﬁrst three (Empty, 10 Tiles and 20 Tiles) evaluate the performance of the agent within its training distribution, while we also include a held-out human designed environment,
Revised: The ﬁrst three (Empty, 10 Tiles and 20 Tiles) evaluate the performance of the agent within its training distribution, while we also include a held-out human designed environment,

Original: LavaCrossing S9N1, ported from MiniGrid Chevalier-Boisvert et al.
Revised: LavaCrossing S9N1, ported from MiniGrid Chevalier-Boisvert et al.

Original: As we see, ACCEL performs best on all of the in sample environments, while also being only one of two approaches to get meaningfully above zero in the human designed task
Revised: As we see, ACCEL performs best on all of the in sample environments, while also being only one of two approaches to get meaningfully above zero in the human designed task.

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: Partially-Observable Navigation
Revised: Partially-Observable Navigation

Original: Next we show the extended results for the MiniGrid experiments
Revised: Next we show the extended results for the MiniGrid experiments.

Original: [Table] We use a series of challenging zero-shot environments (see Fig. 12), introduced in the UED literature (Dennis et al., 2020; Jiang et al., 2021a).
Revised: We use a series of challenging zero-shot environments (see Figure 16), introduced in the UED literature (Dennis et al., 2020; Jiang et al., 2021a).

Original: We include a full table of zero-shot transfer performance for both 10k and 20k PPO updates in Table 3.
Revised: We include the full results in Table 3.

Original: Labyrinth Labyrinth
Revised: [Figure] [Table] [Figure]

Original: Maze
Revised: As we see, both versions of ACCEL signiﬁcantly outperform the baselines. Particularly in the more complex environments like Labyrinth we see large gains vs. the baselines. Also note that PLR outperforms all other baselines, and ACCEL outperforms PLR. We show this in the Table with a statistically signiﬁcant point estimate, but also using the more robust metrics in rliable Agarwal et al.

Original: Maze
Revised: (2021b)

Original: Maze
Revised: , for example the “probability of improvement” shown in Figure 17. Testing the Limits of Current Approaches In the experimental section we showed the results for a large procedurally generated maze, of size 51x51. ACCEL averaged a 53% and 52% success rate when using the empty or DR generator respectively, vs. a next best PLR with 25%.

Original: PerfectMaze* SixteenRooms
Revised: Now we consider going even further.

Original: SixteenRooms
Revised: We tested both versions of ACCEL, as well as DR and PLR, on an even larger maze, this time 101x101, shown in Figure 18. Note that this would be challenging even for human players, since the agent has a partially observable view.

Original: SmallCorridor* LargeCorridor* SimpleCrossing* FourRooms*
Revised: The performance of all methods is signiﬁcantly weaker, with DR and PLR achieving a mean of 4% success rate.

Original: [Table]
Revised: However, ACCEL still outperforms, achieving 7% and 8% success rate with the DR and empty generators respectively. We believe at this point the bottleneck for further improvement may not be the curriculum, but instead the LSTM-based policy which has to remember all previous paths, to solve the maze in the allotted 20,402 steps.

Original: A. A DDITIONAL E XPERIMENTS
Revised: A. A DDITIONAL E XPERIMENTS

Revised: In this section we show a series of additional experiments to understand the performance of ACCEL.

Original: The Sensitivity of Domain Randomization Consider a simple parameterized grid world environment where an agent has to navigate towards a goal.
Revised: The Sensitivity of Domain Randomization Consider a simple parameterized grid world environment where an agent has to navigate towards a goal.

Original: Each level varies in terms of the location of the agent, goal and placement of obstacles.
Revised: Each level varies in terms of the location of the agent, goal and placement of obstacles.

Original: When these obstacles are walls the agent is able to learn quickly, achieving high performance on the training distribution.
Revised: When these obstacles are walls the agent is able to learn quickly, achieving high performance on the training distribution.

Original: However, when these obstacles are lava , which causes instant death on collision, the agent is unable to learn (Fig. 13.a), left).
Revised: However, when these obstacles are lava , which causes instant death on collision, the agent is unable to learn (Figure 19.a), left).

Original: If we change the DR parameterization to not only sample the locations of the lava tiles, but the number of tiles to place , then the agent does indeed learn (Fig. 13.a), right).
Revised: If we change the DR parameterization to not only sample the locations of the lava tiles, but the number of tiles to place , then the agent does indeed learn (Figure 19.a), right).

Original: This subtle change illustrates the sensitivity of DR to the environment parameterization.
Revised: This subtle change illustrates the sensitivity of DR to the environment parameterization.

Original: We refer to this problem as “curse of 
Revised: We refer to this problem as “curse of dimensionality for UED”, whereby increasing the design space leads to a crowding out of levels that are effective for learning. In Figure 19.b) we show the test performance for the three DR agents at the same checkpoint ( updates), as well as a PLR agent using the Binomial DR levels. Despite the poor performance for the

Original: 2000 Student PPO Updates
Revised: 2000 Student PPO Updates

Original: T r a i n i n g R e t u r n
Revised: T r a i n i n g R e t u r n

Original: Lava Walls 2000 Student PPO Updates
Revised: Lava Walls 2000 Student PPO Updates

Original: Fixed Binomial
Revised: Fixed Binomial

Original: Uniform (a) Training Performance 0.0 0.
Revised: Uniform (a) Training Performance 0.0 0.

Original: S o l v e d
Revised: S o l v e d

Original: R a t e
Revised: R a t e

Original: Test: 10Tiles
Revised: Test: 10Tiles

Original: Fixed Bin
Revised: Fixed Bin

Original: Unif PLR (Bin) (b) Test Return 1 Return 0.
Revised: Unif PLR (Bin) (b) Test Return 1 Return 0.

Original: P o s i t i v e
Revised: P o s i t i v e

Original: V a l u e
Revised: V a l u e

Original: L o ss
Revised: L o ss

Original: DR Levels 1 Return 0.
Revised: DR Levels 1 Return 0.

Original: PLR Levels
Revised: PLR Levels

Original: [Figure]
Revised: [Figure]

Original: dimensionality for UED”, whereby increasing the design space leads to a crowding out of levels that are effective for learning.

Original: In Fig.13.b) we show the test performance for the three DR agents at the same checkpoint ( updates), as well as a PLR agent using the Binomial DR levels.

Original: Despite the poor performance for the

Original: DR agent PLR is able to achieve strong performance, outperforming even theUniform DR agent.
Revised: DR agent, PLR is able to achieve strong performance, outperforming even the

Original: To see how this is possible, in Fig. 13.c) we compare the density of levels sampled by DR vs. those in the PLR buffer, sorted by Return and Positive Value Loss.
Revised: To see how this is possible, in Figure 19.c) we compare the density of levels sampled by DR vs. those in the PLR buffer, sorted by Return and Positive Value Loss.

Original: Notably, the levels chosen by PLR are inherently higher return levels, thus, we see that PLR is able to ﬁnd the frontier without handcrafted heuristics.
Revised: Notably, the levels chosen by PLR are inherently higher return levels, thus, we see that PLR is able to ﬁnd the frontier without handcrafted heuristics.

Original: This example illustrates the importance of having a sufﬁcient quantity of frontier levels.
Revised: This example illustrates the importance of having a sufﬁcient quantity of frontier levels.

Original: Indeed, the success of PLR hinges on the ability to reliably sample new levels at the ever-changing frontier of the agent’s capabilities.
Revised: Indeed, the success of PLR hinges on the ability to reliably sample new levels at the ever-changing frontier of the agent’s capabilities.

Original: This is the key motivation for .
Revised: This is the key motivation for ACCEL.

Original: Next, in Fig.14 we show the performance of the three DR agents when tested on each other’s training distributions after one thousand training steps.
Revised: Next, in Figure 20 we show the performance of the three DR agents when tested on each other’s training distributions after one thousand training steps.

Original: All agents perform worst on the Fixed levels, slightly better on the Binomial ones, with the best performance on Uniform.
Revised: All agents perform worst on the Fixed levels, slightly better on the Binomial ones, with the best performance on Uniform.

Original: Notably, even the Fixed agent, the weakest of the three, is able to see almost double the performance on the Uniform levels.
Revised: Notably, even the Fixed agent, the weakest of the three, is able to see almost double the performance on the Uniform levels.

Original: This indicates that a key ingredient to learning potential is having a sufﬁcient number of easier levels to generate learning signal.
Revised: This indicates that a key ingredient to learning potential is having a sufﬁcient number of easier levels to generate learning signal.

Original: Indeed, this was noted in Jiang et al.
Revised: Indeed, this was noted in Jiang et al.

Original: (2021b)
Revised: (2021b)

Original: in their MiniGrid experiments.
Revised: in their MiniGrid experiments.

Original: [Figure]
Revised: [Figure]

Original:  Ablation Studies To investigate the design choices that led to the success of ACCEL, we consider a variety of ablations:
Revised: It is clear Ablation Studies To investigate the design choices that led to the success of ACCEL, we consider a variety of ablations:

Original: • ACCEL The main result included in the paper.
Revised: • ACCEL Using the DR generator.

Original: • Edit Batch
Revised: • Edit Batch

Original: The same ACCEL algorithm but editing the entire replay batch rather than the “easy” levels from the batch.
Revised: The same ACCEL algorithm but editing the entire replay batch rather than the “easy” levels from the batch.

Original: • Learned Editor
Revised: • Learned Editor

Original: The same algorithm changing only the editor, which is optimized for positive value loss.
Revised: The same algorithm changing only the editor, which is optimized for positive value loss.

Original: No Editor This is an ablation on the algorithm structure, we use Algorithm 1 but sample more DR levels instead of editing.
Revised: No Editor This is an ablation on the algorithm structure, we use Algorithm 1 but sample more DR levels instead of editing.

Original: Each of these uses the same exact structure, sampling levels from the DR distribution 10% of the time, and editing levels after replaying from the curator.
Revised: Each of these uses the same exact structure, sampling levels from the DR distribution 10% of the time, and editing levels after replaying from the curator.

Original: For the ﬁrst, we edit the entire replay batch, rather than just the top four easiest (high return minus positive value loss) that we use in the main experiments.
Revised: For the ﬁrst, we edit the entire replay batch, rather than just the top four easiest (high return minus positive value loss) that we use in the main experiments.

Original: The results after 10k updates are shown in Table4.
Revised: The results after 10k updates are shown in Table4.

Original: As we see, Editing the batch levels performs slightly worse than easy, while there is also a decrease in performance for the learned editor.
Revised: As we see, Editing the batch levels performs slightly worse than easy, while there is also a decrease in performance for the learned editor.

Original:  Note however that all of these ablations still outperform the next best baseline (PLR, mean = 0.69).
Revised: [Table] Note however that all of these ablations still outperform the next best baseline (PLR, mean = 0.69).

Original: Finally, the ablation using additional DR levels instead of edited ones performs poorly, showing the strong performance for ACCEL comes from editing rather than the structure of the algorithm [Table] [Table]
Revised: Finally, the ablation using additional DR levels instead of edited ones performs poorly, showing the strong performance for ACCEL comes from editing rather than the structure of the algorithm.

Original: [Figure]
Revised: [Figure]

Original: Diversity of the Level Buffer we compare a buffer of size 4k with no DR sampling against our method with a buffer size of 10k and 10% sampling.
Revised: Diversity of the Level Buffer we compare a buffer of size 4k with no DR sampling against our method with a buffer size of 10k and 10% sampling.

Original: The plots show the proportion of the top 200 levels produced by just ten initial generator levels, with a signiﬁcant increase for the smaller buffer.
Revised: The plots show the proportion of the top 200 levels produced by just ten initial generator levels, with a signiﬁcant increase for the smaller buffer.

Original: We also compare the performance of the two agents on test levels with ten tiles, showing clear outperformance for the lower concentration agent.
Revised: We also compare the performance of the two agents on test levels with ten tiles, showing clear outperformance for the lower concentration agent.

Revised: It is likely that hyperparameters alone will not be sufﬁcient if we want to scale ACCEL to more open-ended problems, which we leave to future work.

Original: B I MPLEMENTATION D ETAILS
Revised: B I MPLEMENTATION D ETAILS

Original: In this section we detail the training procedure for all our experiments.
Revised: In this section we detail the training procedure for all our experiments.

Original: Training for all methods is conducted on a single V100 GPU, using ten CPUs.
Revised: Training for all methods is conducted on a single V100 GPU, using ten CPUs.

Original: The codebase is built on top of open source repos.
Revised: The codebase is built on top of open source repos.

Original: Notably, for PAIRED and Minimax we use  the PyTorch implementation: https://github. com/ucl-dark/paired .
Revised: For PAIRED and Minimax we use results from the authors of Jiang et al. (2021a).

Original: B. E NVIRONMENT D ETAILS
Revised: B. E NVIRONMENT D ETAILS

Original: Learning with Lava The MiniHack environment is an open-source Gym (Brockman et al., 2016) environment which provides a wrapper around the full game of NetHack via the NetHack Learning
Revised: Learning with Lava The MiniHack environment is an open-source Gym (Brockman et al., 2016) environment which provides a wrapper around the full game of NetHack via the NetHack Learning

Original: Environment K¨uttler et al.
Revised: Environment K¨uttler et al.

Original: MiniHack allows users (or agents) the ability the fully specify environments leveraging the full game engine from NetHack.
Revised: MiniHack allows users (or agents) the ability the fully specify environments leveraging the full game engine from NetHack.

Original: For our experiments we use a simple 7x7 grid, and allow the agent to place lava tiles in any location of their choice.
Revised: For our experiments we use a simple 7x7 grid, and allow the agent to place lava tiles in any location of their choice.

Original: We considered three different DR parameterizations in Fig.13.
Revised: We considered three different DR parameterizations in Figure 19.

Original: These are: “Fixed” which always places ten lava blocks, “Binomial” which places twenty blocks, with a 50-50 chance of the block being lava or nothing, and “Uniform” which selects the number of lava blocks to place from the range [0 , 20] .
Revised: These are: “Fixed” which always places ten lava blocks, “Binomial” which places twenty blocks, with a 50-50 chance of the block being lava or nothing, and “Uniform” which selects the number of lava blocks to place from the range [0 , 20] .

Original: The reward is sparse, with the agent receiving +1 for reaching the goal, with a per timestep penalty of 0.01.
Revised: The reward is sparse, with the agent receiving +1 for reaching the goal, with a per timestep penalty of 0.01.

Original: Partially-Observable Navigation
Revised: Partially-Observable Navigation

Original: Each maze consists of a 15 × 15 grid, where each cell can contain a wall, the goal, the agent, or navigable space.
Revised: Each maze consists of a 15 × 15 grid, where each cell can contain a wall, the goal, the agent, or navigable space.

Original: The student agent receives a reward of 1 − T/T max upon reaching the goal, where T is the episode length and T max is the maximum episode length (set to 250).
Revised: The student agent receives a reward of 1 − T/T max upon reaching the goal, where T is the episode length and T max is the maximum episode length (set to 250).

Original: Otherwise, the agent receives a reward of 0 if it fails to reach the goal.
Revised: Otherwise, the agent receives a reward of 0 if it fails to reach the goal.

Original: [Table]
Revised: [Table]

Original: B. E NVIRONMENT D ESIGN P ROCEDURE

Original: The environment design procedure works as follows: at each timestep the adversary agent receives an observation consisting of a map of the entire level and then takes a two dimensional action, consisting of an object and a location, which can be anywhere in the grid.
Revised: The environment design procedure works as follows: at each timestep the adversary agent receives an observation consisting of a map of the entire level and then takes a two dimensional action, consisting of an object and a location, which can be anywhere in the grid.

Original: This is similar to several recent works
Revised: This is similar to several recent works

Original: Dennis et al.
Revised: Dennis et al.

Original: ; Jiang et al.
Revised: ; Jiang et al.

Original: (2021b;a); Khalifa et al.
Revised: (2021b;a); Khalifa et al.

Original: For MiniGrid the object is always a wall.
Revised: For MiniGrid the object is always a wall.

Original: For both methods, the goal and agent location are placed in the ﬁnal two steps.
Revised: For both methods, the goal and agent location are placed in the ﬁnal two steps.

Original: When editing, the editor has ﬁve steps to alter the environment.
Revised: When editing, the editor has ﬁve steps to alter the environment.

Original: For the lava environment we only edit to add or remove walls, while for MiniGrid we allow the editor to also change the goal location.
Revised: For the lava environment we only edit to add or remove lava tiles, while for MiniGrid we allow the editor to also change the goal location.

Original: If lava or walls  are placed in the current location of the goal or agent, then these replace the goal or agent, which must then be relocated in the ﬁnal two steps of the editing episode.
Revised: If lava or walls (or lava) are placed in the current location of the goal or agent, then these replace the goal or agent, which must then be relocated in the ﬁnal two steps of the editing episode.

Original: If the editor attempts to remove the goal or agent location in the ﬁnal two steps then the action is void
Revised: If the editor attempts to remove the goal or agent location in the ﬁnal two steps then the action is void.

Original: . B. H YPERPARAMETERS
Revised: B. H YPERPARAMETERS

Original: The majority of our hyperparameters are inherited from previous works such as Dennis et al.
Revised: The majority of our hyperparameters are inherited from previous works such as Dennis et al.

Original: Jiang et al.
Revised: Jiang et al.

Original: (2021b;a), with a few small changes.
Revised: (2021b;a), with a few small changes.

Original: The ﬁrst, for MiniHack we use the agent model from the NetHack paper (K¨uttler et al., 2020), using the glyphs and blstats as observations.
Revised: The ﬁrst, for MiniHack we use the agent model from the NetHack paper (K ¨ uttler et al., 2020), using the glyphs and blstats as observations.

Original: The agent has both a global and a locally cropped view (produced using the coordinates in the blstats ).
Revised: The agent has both a global and a locally cropped view (produced using the coordinates in the blstats ).

Original: For MiniHack we conduct a grid search across the level replay buffer size { 4000 , 10000 } for both
Revised: For MiniHack we conduct a grid search across the level replay buffer size { 4000 , 10000 } for both

Original: PLR and ACCEL, and for ACCEL
Revised: PLR and ACCEL, and for ACCEL

Original: we sweep across the edit method from { random, positive value loss } where positive value loss equates to a learned editor.
Revised: we sweep across the edit method from { random, positive value loss } where positive value loss equates to a learned editor.

Original: For MiniGrid we maintain the replay buffer size from Jiang et al.
Revised: For MiniGrid we maintain the replay buffer size from Jiang et al.

Original: (2021a)
Revised: (2021a)

Original: and only conduct the ACCEL grid search over the edit objective, again running both { random, positive value loss } , as well as the levels to edit from { easy, batch }  .
Revised: and only conduct the ACCEL grid search over the edit objective, again running both { random, positive value loss } , as well as the levels to edit from { easy, batch } and replay rate from { 0.8, 0.9 } .

Revised: For MiniGrid, we follow the protocol from Jiang et al.

Revised: (2021a)

Revised: and select the best hyperparameters using the validation levels { 16Rooms, Labyrinth, Maze } .

Original: The ﬁnal hyperparameters chosen are shown in Table 6.
Revised: The ﬁnal hyperparameters chosen are shown in Table 6.

Original: [Table]
Revised: [Table]