Original: Deep Surrogate Assisted Generation of Environments
Revised: Deep Surrogate Assisted Generation of Environments

Original: Anonymous Author(s)
Revised: Anonymous Author(s)

Original: AffiliationAddress email
Revised: AffiliationAddress email

Original: Abstract
Revised: Abstract

Original: Recent progress in reinforcement learning (RL) has started producing generallycapable agents that can solve a distribution of complex environments.
Revised: Recent progress in reinforcement learning (RL) has started producing generallycapable agents that can solve a distribution of complex environments.

Original: These agentsare typically tested on fixed, human-authored environments.
Revised: These agentsare typically tested on fixed, human-authored environments.

Original: On the other hand,quality diversity (QD) optimization has been proven to be an effective component ofenvironment generation algorithms, which can generate collections of high-qualityenvironments that are diverse in the resulting agent behaviors.
Revised: On the other hand,quality diversity (QD) optimization has been proven to be an effective component ofenvironment generation algorithms, which can generate collections of high-qualityenvironments that are diverse in the resulting agent behaviors.

Original: However, thesealgorithms require potentially expensive simulations of agents on newly generatedenvironments.
Revised: However, thesealgorithms require potentially expensive simulations of agents on newly generatedenvironments.

Original: We propose Deep Surrogate Assisted Generation of Environments(DSAGE), a sample-efficient QD environment generation algorithm that maintainsa deep surrogate model for predicting agent behaviors in new environments.
Revised: We propose Deep Surrogate Assisted Generation of Environments(DSAGE), a sample-efficient QD environment generation algorithm that maintainsa deep surrogate model for predicting agent behaviors in new environments.

Original: Resultsin two benchmark domains show that DSAGE significantly outperforms existing
Revised: Resultsin two benchmark domains show that DSAGE significantly outperforms existing

Original: QD environment generation algorithms in discovering collections of environmentsthat elicit diverse behaviors of a state-of-the-art RL agent and a planning agent.
Revised: QD environment generation algorithms in discovering collections of environmentsthat elicit diverse behaviors of a state-of-the-art RL agent and a planning agent.

Original:  Introduction
Revised:  Introduction

Original: We present an efficient method of automatically generating a collection of environments that elicitdiverse agent behaviors.
Revised: We present an efficient method of automatically generating a collection of environments that elicitdiverse agent behaviors.

Original: As a motivating example, consider deploying a robot agent at scale in avariety of home environments.
Revised: As a motivating example, consider deploying a robot agent at scale in avariety of home environments.

Original: The robot should generalize by performing robustly not only in testhomes, but in any end user’s home.
Revised: The robot should generalize by performing robustly not only in testhomes, but in any end user’s home.

Original: To enable such generalization, the test environments shouldhave good coverage for the robot agent.
Revised: To enable such generalization, the test environments shouldhave good coverage for the robot agent.

Original: However, obtaining such coverage may be difficult, as thegenerated environments would depend on the application domain, e.g. kitchen or living room, and onthe specific agent we want to test, since different agents exhibit different behaviors.
Revised: However, obtaining such coverage may be difficult, as thegenerated environments would depend on the application domain, e.g. kitchen or living room, and onthe specific agent we want to test, since different agents exhibit different behaviors.

Original: To enable generalization of autonomous agents to new environments with differing levels of complexity, previous work on open-ended learning [1, 2] has integrated the environment generation and theagent training processes.
Revised: To enable generalization of autonomous agents to new environments with differing levels of complexity, previous work on open-ended learning [1, 2] has integrated the environment generation and theagent training processes.

Original: The interplay between the two processes acts as a natural curriculum for theagents to learn robust skills that generalize to new, unseen environments [3–5].
Revised: The interplay between the two processes acts as a natural curriculum for theagents to learn robust skills that generalize to new, unseen environments [3–5].

Original: The performance ofthese agents has been evaluated either in environments from the training distribution [1, 2, 5] or insuites of manually authored environments [3,6, 4].
Revised: The performance ofthese agents has been evaluated either in environments from the training distribution [1, 2, 5] or insuites of manually authored environments [3, 4].

Original: As a step towards testing generalizable agents, there has been increasing interest in competitions [7, 8]that require agents to generalize to new game layouts.
Revised: As a step towards testing generalizable agents, there has been increasing interest in competitions [6, 7]that require agents to generalize to new game layouts.

Original: Despite the recent progress of deep learningagents in fixed game domains, e.g. in Chess [9], Go [10], Starcraft [11], and Poker [12, 13], it hasbeen rule-based agents that have succeeded in these competitions [8].
Revised: Despite the recent progress of deep learningagents in fixed game domains, e.g. in Chess [8], Go [9], Starcraft [10], and Poker [11, 12], it hasbeen rule-based agents that have succeeded in these competitions [7].

Original: Such competitions also rely onmanually authored game levels as a test set, handcrafted by a human designer.
Revised: Such competitions also rely onmanually authored game levels as a test set, handcrafted by a human designer.

Original: While manually authored environments are important for standardized testing, creating these environments can be tedious and time-consuming.
Revised: While manually authored environments are important for standardized testing, creating these environments can be tedious and time-consuming.

Original: Additionally, manually authored test suites are ofteninsufficient for eliciting the diverse range of possible agent behaviors.
Revised: Additionally, manually authored test suites are ofteninsufficient for eliciting the diverse range of possible agent behaviors.

Original: Instead, we would like aninteractive test set that proposes an environment, observes the agent’s performance and behavior, and
Revised: Instead, we would like aninteractive test set that proposes an environment, observes the agent’s performance and behavior, and

Original: Sample Solution
Revised: Sample Latent Code

Original: UpdateSurrogateArchive
Revised: UpdateSurrogateArchive

Original: Extract Predicted Agent Performance and
Revised: Extract Predicted Agent Performance and

Original: Measured Behavior
Revised: Measured Behavior

Original: Generate Environments from Solutions and Evaluate
Revised: Generate Environments from Latent Codes and Evaluate

Original: Extract Ground Truth Agent Performance and Behavior Data
Revised: Extract Ground Truth Agent Performance and Behavior Data

Original: Downsample Archive
Revised: Downsample Archive

Original: Update Dataset
Revised: Update Dataset

Original: Update Ground Truth Archive
Revised: Update Ground Truth Archive

Original: Generate Environment
Revised: Generate Environment

Original: Train the Surrogate Model on the Dataset
Revised: Train the Surrogate Model on the Dataset

Original: Environment Generation
Revised: Environment Generation

Original: Occupancy Prediction
Revised: Occupancy Prediction

Original: Stack with Occupancy Grid
Revised: Stack with Occupancy Grid

Original: Self-supervised Agent Occupancy Prediction
Revised: Self-supervised Agent Occupancy Prediction

Original: A
Revised: A

Original: A
Revised: [Figure]

Original: Model Exploitation
Revised: Model Exploitation

Original: Agent Simulation
Revised: Agent Simulation

Original: Model Improvement then proposes new environments that diversify the agent behaviors, based on what the system haslearned from previous execution traces of the agent.
Revised: Model Improvement then proposes new environments that diversify the agent behaviors, based on what the system haslearned from previous execution traces of the agent.

Original: To address collecting environments with diverse agent behaviors, prior work frames the problemas a quality diversity (QD) problem [15–17].
Revised: To address collecting environments with diverse agent behaviors, prior work frames the problemas a quality diversity (QD) problem [13–15].

Original: A QD problem consists of an objective function, e.g.whether the agent can solve the environment, and measure functions, e.g. how long the agent takesto complete their task.
Revised: A QD problem consists of an objective function, e.g.whether the agent can solve the environment, and measure functions, e.g. how long the agent takesto complete their task.

Original: The measure functions quantify the behavior we would like to vary in theagent, allowing practitioners to specify the case coverage they would like to see in the domain theyare testing.
Revised: The measure functions quantify the behavior we would like to vary in theagent, allowing practitioners to specify the case coverage they would like to see in the domain theyare testing.

Original: While QD algorithms can generate diverse collections of environments, they require alarge number of environment evaluations to produce the collection, and each of these evaluationsrequires multiple time-consuming simulated executions of potentially stochastic agent policies.
Revised: While QD algorithms can generate diverse collections of environments, they require alarge number of environment evaluations to produce the collection, and each of these evaluationsrequires multiple time-consuming simulated executions of potentially stochastic agent policies.

Original: We study how deep surrogate models that predict agent performance can accelerate the generationof environments that are diverse in agent behaviors .
Revised: We study how deep surrogate models that predict agent performance can accelerate the generationof environments that are diverse in agent behaviors .

Original: We draw upon insights from model-basedquality diversity algorithms that have been previously shown to improve sample efficiency in designoptimization [18] and Hearthstone deckbuilding [19].
Revised: We draw upon insights from model-basedquality diversity algorithms that have been previously shown to improve sample efficiency in designoptimization [16] and Hearthstone deckbuilding [17].

Original: Environments present a much more complexprediction task because the evaluation of environments involves simulating stochastic agent policies,and small changes in the environment may result in large changes in the emergent agent behaviors [20].
Revised: Environments present a much more complexprediction task because the evaluation of environments involves simulating stochastic agent policies,and small changes in the environment may result in large changes in the emergent agent behaviors [18].

Original: We make the following contributions: (1) We propose the use of deep surrogate models to predictagent performance in new environments.
Revised: We make the following contributions: (1) We propose the use of deep surrogate models to predictagent performance in new environments.

Original: Our algorithm, Deep Surrogate Assisted Generation of
Revised: Our algorithm, Deep Surrogate Assisted Generation of

Original: Environments (DSAGE) (Fig. 1), integrates deep surrogate models into quality diversity optimizationto efficiently generate diverse environments.
Revised: Environments (DSAGE) (Fig. 1), integrates deep surrogate models into quality diversity optimizationto efficiently generate diverse environments.

Original: (2) We show in two benchmark domains from previous work, a Maze domain [3, 4] with a trained ACCEL agent [4] and a Mario domain [21, 16] with an
Revised: (2) We show in two benchmark domains from previouswork, a Maze domain [3, 4] with a trained ACCEL agent [4] and a Mario domain [19, 14] with an

Original: A* agent [22], that DSAGE outperforms state-of-the-art QD algorithms in discovering diverse agentbehaviors.
Revised: A* agent [20], that DSAGE outperforms state-of-the-art QD algorithms in discovering diverse agentbehaviors.

Original: (3) We show with ablation studies that training the surrogate model with ancillary agentbehavior data and downsampling a subset of solutions from the surrogate archive results in substantialimprovements in performance, compared to the surrogate models of previous work [19].
Revised: (3) We show with ablation studies that training the surrogate model with ancillary agent behavior data and downsampling a subset of solutions from the surrogate archive results in substantialimprovements in performance, compared to the surrogate models of previous work [17].

Original:  Problem Definition
Revised:  Problem Definition

Original: Quality diversity (QD) optimization.
Revised: Quality diversity (QD) optimization.

Original: We adopt the QD problem definition from previous work [23].
Revised: We adopt the QD problem definition from previous work [21].

Original: A QD optimization problem specifies an objective function f : R n → R and a joint measure functionm : R n → R m .
Revised: A QD optimization problem specifies an objective function f : R n → R and a joint measure functionm : R n → R m .

Original: For each element s ∈ S , where S ⊆
Revised: For each element s ∈ S , where S ⊆

Original: R m is the range of the measure function, the
Revised: R m is the range of the measure function, the

Original: QD goal is to find a solution θ ∈ R n such that m ( θ )
Revised: QD goal is to find a solution θ ∈ R n such that m ( θ )

Original: =
Revised: =

Original: s and f ( θ ) is maximized.
Revised: s and f ( θ ) is maximized.

Original: Since the range of the measure function can be continuous, we restrict ourselves to algorithms fromthe MAP-Elites family [24, 25] that discretize this space into a finite number of M cells.
Revised: Since the range of the measure function can be continuous, we restrict ourselves to algorithms fromthe MAP-Elites family [22, 23] that discretize this space into a finite number of M cells.

Original: A solutionθ is mapped to a cell based on its measure m ( θ ) .
Revised: A solutionθ is mapped to a cell based on its measure m ( θ ) .

Original: The solutions that occupy cells form an archive ofsolutions.
Revised: The solutions that occupy cells form an archive ofsolutions.

Original: Our goal is to find solutions θ i , i ∈ { 1 , ..., M } that maximize the objective f for all cellsin the measure space.
Revised: Our goal is to find solutions θ i , i ∈ { 1 , ..., M } that maximize the objective f for all cellsin the measure space.

Original: [Equation]
Revised: [Equation]

Original: The computed sum in Eq.
Revised: The computed sum in Eq.

Original: 1 is defined as the QD-score [26], where empty cells have an objectivevalue of 0.
Revised: 1 is defined as the QD-score [24], where empty cells have an objectivevalue of 0.

Original: A second metric of the performance of a QD algorithm is coverage of the measure space,defined as the proportion of cells that are filled in by solutions: 1 M (cid:80) Mi
Revised: A second metric of the performance of a QD algorithm is coverage of the measure space,defined as the proportion of cells that are filled in by solutions: 1 M (cid:80) Mi

Original: =1 1 θ i .
Revised: =1 1 θ i .

Original: QD for environment generation.
Revised: QD for environment generation.

Original: We assume a single agent acting in an environment parameterizedby θ ∈ R n .
Revised: We assume a single agent acting in an environment parameterizedby θ ∈ R n .

Original: The environment parameters can be locations of different objects or latent variables thatare passed as inputs to a generative model [27].
Revised: The environment parameters can be locations of different objects or latent variables thatare passed as inputs to a generative model [25].

Original: 1 A QD algorithm generates new solutions θ andevaluates them by simulating the agent on the environment parameterized by θ .
Revised: 1 A QD algorithm generates new solutions θ andevaluates them by simulating the agent on the environment parameterized by θ .

Original: The evaluation returnsan objective value f and measure values m .
Revised: The evaluation returnsan objective value f and measure values m .

Original: The QD algorithm attempts to generate environmentsthat maximize f but are diverse with respect to the measures m .
Revised: The QD algorithm attempts to generate environmentsthat maximize f but are diverse with respect to the measures m .

Original:  Background and Related Work
Revised:  Background and Related Work

Original: Quality diversity (QD) optimization.
Revised: Quality diversity (QD) optimization.

Original: QD optimization originated in the genetic algorithm community with diversity optimization [28], the predecessor to QD.
Revised: QD optimization originated in the genetic algorithm community with diversity optimization [26], the predecessor to QD.

Original: Later work introduced objectives todiversity optimization and resulted in the first QD algorithms: Novelty Search with Local Competition [29] and MAP-Elites [25, 24].
Revised: Later work introduced objectives todiversity optimization and resulted in the first QD algorithms: Novelty Search with Local Competition [27] and MAP-Elites [23, 22].

Original: The QD community has grown beyond its genetic algorithmroots, with algorithms being proposed based on gradient ascent [23], Bayesian optimization [30],differential evolution [31], and evolution strategies [14, 32, 33].
Revised: The QD community has grown beyond its genetic algorithmroots, with algorithms being proposed based on gradient ascent [21], Bayesian optimization [28],differential evolution [29], and evolution strategies [30–32].

Original: QD algorithms have applications indamage recovery in robotics [24], reinforcement learning [34–36], and generative design [18, 37].
Revised: QD algorithms have applications indamage recovery in robotics [22], reinforcement learning [33–35], and generative design [16, 36].

Original: Among the QD algorithms, those of particular interest to us are the model-based ones.
Revised: Among the QD algorithms, those of particular interest to us are the model-based ones.

Original: Currentmodel-based [38, 39] QD algorithms either (1) learn a surrogate model of the objective and measurefunctions [18, 40, 41], e.g. a Gaussian process or neural network, (2) learn a generative modelof the representation parameters [42, 43], or (3) draw inspiration from model-based RL [44, 45].
Revised: Currentmodel-based [37, 38] QD algorithms either (1) learn a surrogate model of the objective and measurefunctions [16, 39, 40], e.g. a Gaussian process or neural network, (2) learn a generative modelof the representation parameters [41, 42], or (3) draw inspiration from model-based RL [43, 44].

Original: In particular, Deep Surrogate Assisted MAP-Elites (DSA-ME)
Revised: In particular, Deep Surrogate Assisted MAP-Elites (DSA-ME)

Original: [19] trains a deep surrogate modelon a diverse dataset of solutions generated by MAP-Elites and then leverages the model to guide
Revised: [17] trains a deep surrogate modelon a diverse dataset of solutions generated by MAP-Elites and then leverages the model to guide

Original: MAP-Elites.
Revised: MAP-Elites.

Original: However, DSA-ME has only been applied to Hearthstone deck building, a simplerprediction problem than predicting agent behavior in generated environments.
Revised: However, DSA-ME has only been applied to Hearthstone deck building, a simplerprediction problem than predicting agent behavior in generated environments.

Original: Additionally, DSAME is specific to MAP-Elites only and cannot run other QD algorithms to exploit the surrogatemodel.

Original: Furthermore, DSA-ME is restricted to direct search and cannot integrate generative models togenerate environments that match a provided dataset.

Original: Automatic environment generation.
Revised: Automatic environment generation.

Original: Automatic environment generation algorithms have beenproposed in a variety of fields.
Revised: Automatic environment generation algorithms have beenproposed in a variety of fields.

Original: Methods between multiple communities often share generation techniques, but differ in how each community applies the generation algorithms.
Revised: Methods between multiple communities often share generationtechniques, but differ in how each community applies the generation algorithms.

Original: For example, in the procedural content generation (PCG) field [46], an environment generatorproduces video game levels that result in player enjoyment.
Revised: For example, in the procedural content generation (PCG) field [45], an environment generatorproduces video game levels that result in player enjoyment.

Original: Since diversity of player experienceand game mechanics is valued in games, many level generation systems incorporate QD optimization [47, 16, 48–52].
Revised: Since diversity of player experienceand game mechanics is valued in games, many level generation systems incorporate QD optimization [46, 14, 47–51].

Original: The procedural content generation via machine learning (PCGML) [53, 54]subfield studies environment generators that incorporate machine learning techniques such as Markov
Revised: The procedural content generation via machine learning (PCGML) [52, 53]subfield studies environment generators that incorporate machine learning techniques such as Markov

Original: Chains [55], probabilistic graphical models [56], LSTMs [57], generative models [58–61], andreinforcement learning [62, 63].
Revised: Chains [54], probabilistic graphical models reinforcement learning [61, 62].

Original: Prior work [64] has leveraged surrogate models trained on offlinedata to accelerate search-based PCG [65].
Revised: Prior work [63] has leveraged surrogate models trained on offlinedata to accelerate search-based PCG [64].

Original: Environment generation methods have also been proposed by the scenario generation community inrobotics.
Revised: Environment generation methods have also been proposed by the scenario generation community inrobotics.

Original: Early work explored automated methods for generating road layouts, vehicle arrangements,and vehicle behaviors for testing autonomous vehicles [66–72].
Revised: Early work explored automated methods for generating road layouts, vehicle arrangements,and vehicle behaviors for testing autonomous vehicles [65–71].

Original: Outside of autonomous vehicles,prior work [73] evaluates robot motion planning algorithms by generating environments that targetspecific motion planning behaviors.
Revised: Outside of autonomous vehicles,prior work [72] evaluates robot motion planning algorithms by generating environments that targetspecific motion planning behaviors.

Original: In human-robot interaction, QD algorithms have been applied asenvironment generators to find failures in shared autonomy systems [17] and human-aware plannerstested in the collaborative Overcooked domain [15]
Revised: In human-robot interaction, QD algorithms have been applied asenvironment generators to find failures in shared autonomy systems [15] and human-aware plannerstested in the collaborative Overcooked domain [13]

Original: Environment generation can also help improve the generality of RL agents.
Revised: Environment generation can also help improve the generality of RL agents.

Original: Prior work proposesdirectly applying PCG level generation algorithms to improve the robustness of RL [74, 75] or tobenchmark RL agents [76].
Revised: Prior work proposesdirectly applying PCG level generation algorithms to improve the robustness of RL [73, 74] or tobenchmark RL agents [75].

Original: Paired Open-ended Trailblazer (POET)
Revised: Paired Open-ended Trailblazer (POET)

Original: [1, 2] coevolves a population ofboth agents and environments to discover specialized agents that solve complex tasks.
Revised: [1, 2] coevolves a population ofboth agents and environments to discover specialized agents that solve complex tasks.

Original: POET inspireda variety of open-ended coevolution algorithms [77–79, 5].
Revised: POET inspireda variety of open-ended coevolution algorithms [76–78, 5].

Original: Later work proposes the PAIRED [3], PLR [80, 6], and ACCEL [4] algorithms that train a single generally capable agent by maximizingthe regret between a pair of agents.
Revised: Later work proposes the PAIRED [3]and ACCEL [4] algorithms that train a single generally capable agent by maximizing the regretbetween a pair of agents.

Original: These methods generate environments in parallel withanagent to create an  automatic training curriculum.
Revised: These methods generate environments in parallel with an agent to createan automatic training curriculum.

Original: However, the authors validate these methods onhuman-designed environments [81].
Revised: However, the authors validate these methods on human-designedenvironments [79].

Original: Our work proposes a method that automatically generates validenvironments that reveal diverse behaviors of these more general RL agents.
Revised: Our work proposes a method that automatically generates valid environments thatreveal diverse behaviors of these more general RL agents.

Original:  Deep Surrogate Assisted Generation of Environments (DSAGE)
Revised:  Deep Surrogate Assisted Generation of Environments (DSAGE)

Original: Algorithm.
Revised: Algorithm.

Original: We propose the Deep Surrogate Assisted Generation of Environments (DSAGE) algorithm for discovering environments that elicit diverse agent behaviors.
Revised: We propose the Deep Surrogate Assisted Generation of Environments (DSAGE) algorithm for discovering environments that elicit diverse agent behaviors.

Original: Akin to the MAP-Elites familyof QD algorithms, DSAGE maintains a ground-truth archive where solutions are stored based ontheir ground-truth evaluations.
Revised: Akin to the MAP-Elites familyof QD algorithms, DSAGE maintains a ground-truth archive where solutions are stored based ontheir ground-truth evaluations.

Original: Simultaneously, DSAGE also trains and exploits a deep surrogatemodel for predicting the behavior of a fixed agent in new environments.
Revised: Simultaneously, DSAGE also trains and exploits a deep surrogatemodel for predicting the behavior of a fixed agent in new environments.

Original: The QD optimization occursin three phases that take place in an outer loop: model  exploitation,agent simulation, and modelimprovement (Fig. 1).
Revised: The QD optimization occursin three phases that take place in an outer loop: model improvement, model exploitation, and agentsimulation (Fig. 1).

Original: Algorithm 1 provides the pseudocode for the DSAGE algorithm.
Revised: Algorithm 1 provides the pseudocode for the DSAGE algorithm.

Original: The model exploitation phase (lines 11–20) is an inner loop that leverages existing QD optimizationalgorithms and the predictions of the deep surrogate model to build an archive – referred to as thesurrogate archive – of solutions.
Revised: The model exploitation phase (lines 11–20) is an inner loop that leverages existing QD optimizationalgorithms and the predictions of the deep surrogate model to build an archive – referred to as thesurrogate archive – of solutions.

Original: The first step of this phase is to query a list of B candidate solutionsthrough the QD algorithm’s ask method.
Revised: The QD algorithm returns a list of B candidate solutions throughthe ask method.

Original: These solutions are environment parameters, e.g., latentvectors of a GAN, which are passed through the environment generator, e.g., a GAN, to create anenvironment (line 15).
Revised: These solutions are environment parameters, e.g., latent vectors of a GAN, whichare passed through the environment generator, e.g., a GAN, to create an environment (line 15).

Original: Next, we make predictions with the surrogate model. The surrogate modelfirst predicts data representing the agent’s behavior, e.g., the probability of occupying each discretizedtile in the environment (line 16), referred to as “ancillary agent behavior data” ( y ). The predictedancillary agent behavior data ( ˆ y ) then guides the surrogate model’s downstream prediction of theobjective ( ˆ f ) and the measure values( ˆ m ) (line 17).
Revised: The surrogate model predicts ancillary agent behavior data (line 16), which guides its downstreamprediction of the objective and the measure values (line 17).

Original: Finally, the QD algorithm’s tell method addsthesolution to the surrogate archive based on the predicted objective and measure values.Note that since DSAGE is independent of the QD algorithm, the ask and tell methods abstract out the QD algorithm’s details. For example, when the QD algorithm is MAP-Elites or CMA-ME, tell addssolutions if the cell in the measure space that they belong to is empty or if the existing solution inthat cell has a lower objective value.
Revised: The tell method adds the solution to thesurrogate archive based on the predicted objective and measure values.

Original: For CMA-ME, tell also includes updating internal CMA-ESparameters.

Original: The agent simulation phase (lines 21–28) inserts a subset of solutions from the surrogate archive intothe ground-truth archive.
Revised: The agent simulation phase (lines 21–28) inserts a subset of solutions from the surrogate archive intothe ground-truth archive.

Original: This phase begins by selecting the subset of solutions from the surrogatearchive (line 21).
Revised: This phase begins by selecting the subset of solutions from the surrogatearchive (line 21).

Original: The selected solutions are evaluated by generating the corresponding environment(line 23) and simulating a fixed agent to obtain the true objective and measure values, as well as ancillary agent behavior data (line 24).
Revised: The selected solutions are evaluated by generating the corresponding environment(line 23) and simulating a fixed agent to obtain the true objective and measure values, as well asancillary agent behavior data (line 24).

Original: Evaluation data is appended to the dataset, and solutions thatimprove their corresponding cell in the ground-truth archive are added to that archive (lines 25, 26).
Revised: Evaluation data is appended to the dataset, and solutions thatimprove their corresponding cell in the ground-truth archive are added to that archive (lines 25, 26).

Original: In the model improvement phase (line 29), the surrogate model is trained in a self-supervised mannerthrough the supervision provided by the agent simulations and the ancillary agent behavior data.
Revised: In the model improvement phase (line 29), the surrogate model is trained in a self-supervised mannerthrough the supervision provided by the agent simulations and the ancillary agent behavior data.

Original: The algorithm is initialized by generating random solutions and simulating the agent in the corresponding environments (lines 2-8).
Revised: The algorithm is initialized by generating random solutions and simulating the agent in the corresponding environments (lines 2-8).

Original: Subsequently, every outer iteration (lines 10-30) consists of modelexploitation followed by agent simulation and ending with model improvement.do 8 end do do do end
Revised: Subsequently, every outer iteration (lines 10-30) consists of modelexploitation followed by agent simulation and ending with model improvement.

Original: Model Exploitation

Original: Agent Simulation

Original: Model Improvement 30 end

Original: Self-supervised prediction of ancillary agent behavior data.
Revised: Self-supervised prediction of ancillary agent behavior data.

Original: By default, a surrogate modeldirectly predicts the objective and measure values based on the initial state of the environment andthe agent (provided in the form of a one-hot encoded image).
Revised: By default, a surrogate modeldirectly predicts the objective and measure values based on the initial state of the environment andthe agent (provided in the form of a one-hot encoded image).

Original: However, we anticipate that directprediction will be challenging in some domains, as it requires understanding the agent’s trajectory inthe environment.
Revised: However, we anticipate that directprediction will be challenging in some domains, as it requires understanding the agent’s trajectory inthe environment.

Original: Thus, we provide additional supervision to the surrogate model in DSAGE via atwo-stage self-supervised process.
Revised: Thus, we provide additional supervision to the surrogate model in DSAGE via atwo-stage self-supervised process.

Revised: Input: N : Maximum number of evaluations, n rand : Number of initial random solutions,

Revised: N exploit : Number of iterations in the model exploitation phase, B : Batch size for the model exploitation QD optimizer

Revised: Output: Final version of the ground-truth archive A gt 1 Initialize the ground-truth archive

Revised: A gt , the dataset D , and the deep surrogate model sm 2 Θ ← generate_random_solutions ( n rand ) 3 for θ ∈ Θ do env ← g ( θ ) f, m , y ← evaluate ( env )

Revised: D ← D ∪ ( θ , f, m , y )

Original: First, a deep neural network predicts ancillary agent behavior data.
Revised: A gt ← add_solution ( A gt , ( θ , f, m ) ) 8 end 9 evals ← n rand 10 while evals < N do Initialize a QD optimizer qd with the surrogate archive A surrogate for itr ∈ { 1 , 2 , . . , N exploit } do Θ ← qd.ask ( B ) for θ ∈ Θ do env ← g ( θ ) ˆ y ← sm.predict_ancillary ( env ) ˆ f, ˆ m ← sm.predict ( env, ˆ y ) qd.tell

Revised: ( θ , ˆ f, ˆ m ) end end Θ ← select_solutions ( A surrogate ) for θ ∈ Θ do env ← g ( θ ) f, m , y ← evaluate ( env )

Revised: D ← D ∪ ( θ , f, m , y )

Revised: A gt ← add_solution ( A gt , ( θ , f, m ) )

Revised: evals ← evals + end sm.train

Revised: ( D )

Revised: Model Exploitation

Revised: Agent Simulation

Revised: Model Improvement 30 end

Revised: First, a deep neural network predicts ancillary agent behavior data.

Original: In our work, we obtain this databy recording the expected number of times the agent visits each discretized tile in the environment,resulting in an “occupancy grid.”
Revised: In our work, we obtain this databy recording the expected number of times the agent visits each discretized tile in the environment,resulting in an “occupancy grid.”

Original: We then concatenate the predicted ancillary information, i.e., thepredicted occupancy grid, with the one-hot encoded image of the environment and pass them throughanother deep neural network to obtain the predicted objective and measure values.
Revised: We then concatenate the predicted ancillary information, i.e., thepredicted occupancy grid, with the one-hot encoded image of the environment and pass them throughanother deep neural network to obtain the predicted objective and measure values.

Original: We use CNNs for both predictors and include details about the architecture in Appendix B.
Revised: We use CNNs forboth predictors and include details about the architecture in Appendix B.

Original: As a baseline, we compareour model with a CNN that directly predicts the objective and measure values without the help ofancillary data.
Revised: As a baseline, we compareour model with a CNN that directly predicts the objective and measure values without the help ofancillary data.

Original: Downsampling to select solutions from the surrogate archive.
Revised: Downsampling to select solutions from the surrogate archive.

Original: After the model exploitation phase,the surrogate archive is populated with solutions that were predicted to be high-performing anddiverse.
Revised: After the model exploitation phase,the surrogate archive is populated with solutions that were predicted to be high-performing anddiverse.

Original: Hence, a basic selection mechanism (line 21) would select all solutions from the surrogatearchive, identical to DSA-ME [19].
Revised: Hence, a basic selection mechanism (line 21) would select all solutions from the surrogatearchive, identical to DSA-ME [17].

Original: However, if the surrogate archive is overly populated, fullselection may result in a large number of ground-truth evaluations per outer-loop iteration, leadingto fewer outer loops and less surrogate model training.
Revised: However, if the surrogate archive is overly populated, fullselection may result in a large number of ground-truth evaluations per outer-loop iteration, leadingto fewer outer loops and less surrogate model training.

Original: To balance the trade-off between evaluatingsolutions from the surrogate archive and training the surrogate model, we only select a subset ofsolutions for evaluation by downsampling the surrogate archive.
Revised: To balance the trade-off between evaluatingsolutions from the surrogate archive and training the surrogate model, we only select a subset ofsolutions for evaluation by downsampling the surrogate archive.

Original: Downsampling uniformly dividesthe surrogate archive into sub-regions of cells and selects a random solution from each area.
Revised: Downsampling uniformly dividesthe surrogate archive into sub-regions of cells and selects a random solution from each area.

Original:  Domains
Revised:  Domains

Original:  We test our algorithms in two benchmark domains from previous work: a Maze domain [3, 4] witha trained ACCEL agent [4] and a Mario domain [82, 16] with an A* agent [22].
Revised: [Figure] We test our algorithms in two benchmark domains from previous work: a Maze domain [3, 4] witha trained ACCEL agent [4] and a Mario domain [80, 14] with an A* agent [20].

Original: We select thesedomains because, despite their relative simplicity (each environment is represented as a 2D grid oftiles), agents in these environments exhibit complex and diverse behaviors.
Revised: We select thesedomains because, despite their relative simplicity (each environment is represented as a 2D grid oftiles), agents in these environments exhibit complex and diverse behaviors.

Original: Appendix A containsadditional details about the environments.
Revised: Appendix A containsadditional details about the environments.

Original: Maze.
Revised: Maze.

Original: The environment generator directly outputs a 16 × 16 map of walls and empty cells whichdefine a MiniGrid environment [83].
Revised: The environment generator directly outputs a 16 × 16 map of walls and empty cells.

Original: The starting cell and goal cell of the maze are set to the pairof reachable empty cells that are furthest apart in this map, as identified by the Floyd-Warshallalgorithm [84].
Revised: Thestarting cell and goal cell of the maze are set to the pair of reachable empty cells that are furthestapart in this map, as identified by the Floyd-Warshall algorithm [81].

Original: The agent needs to reach the goal cell from the starting cell within a specified timelimit.
Revised: The agent needs to reach thegoal cell from the starting cell within a specified time limit.

Original: We select the best trained ACCEL [4] agent out of four random seeds.
Revised: We select the best trained ACCEL [4]agent out of four random seeds.

Original: Since we wish to generate valid environments, we set a binary objective function f that is 1 ifthe generated environment is solvable, i.e., if the goal cell is reachable by the starting cell, and 0otherwise.
Revised: Since we wish to generate valid environments, we set a binary objective function f that is 1 ifthe generated environment is solvable, i.e., if the goal cell is reachable by the starting cell, and 0otherwise.

Original: Furthermore, we wish to generate environments that vary with respect to the challengethey provide to the agent, thus we selected as measures (1) number of wall cells (ranges from 0 to256), and (2) mean agent path length (ranges from 0 to 648, where 648 indicates a failure to reachthe goal).
Revised: Furthermore, we wish to generate environments that vary with respect to the challengethey provide to the agent, thus we selected as measures (1) number of wall cells (ranges from 0 to256), and (2) mean agent path length (ranges from 0 to 648, where 648 indicates a failure to reachthe goal).

Original: Since the ACCEL agent’s policy is stochastic, we average the path length over 50 episodes(validity of the level and the number of wall cells do not depend on the agent).
Revised: Since the ACCEL agent’s policy is stochastic, we average the path length over 50 episodes(validity of the level and the number of wall cells do not depend on the agent).

Original: Mario.
Revised: Mario.

Original: Drawing upon prior work [16, 58], we generate environments for the Mario AI Framework [85,21] by passing latent vectors to a GAN which was pre-trained on human-authored levels.
Revised: Drawing upon prior work [14, 57], we generate environments for the Mario AI Framework [82,19] by passing latent vectors to a GAN which was pre-trained on human-authored levels.

Original: Each leveloutput by the GAN’s generator is a 16 × 56 grid of tiles, where each tile can be one of 17 differentobjects.
Revised: Each leveloutput by the GAN’s generator is a 16 × 56 grid of tiles, where each tile can be one of 17 differentobjects.

Original: In each environment, we run the A* agent developed by Robin Baumgarten [22].
Revised: In each environment, we run the A* agent developed by Robin Baumgarten [20].

Original: Since we wish to generate playable levels, we set the objective as the completion rate , i.e., theproportion of the level that the agent completes before dying.
Revised: Since we wish to generate playable levels, we set the objective as the completion rate , i.e., theproportion of the level that the agent completes before dying.

Original: We additionally want to generateenvironments that result in qualitatively different agent behaviors, thus we selected as measures: (1)sky tiles , the number of tiles of a certain type that are in the top half of the 2D grid (ranges from 0 to150), (2) number of jumps , the number of times that the A* agent jumps during its execution (rangesfrom 0 to 100).
Revised: We additionally want to generateenvironments that result in qualitatively different agent behaviors, thus we selected as measures: (1)sky tiles , the number of tiles of a certain type that are in the top half of the 2D grid (ranges from 0 to150), (2) number of jumps , the number of times that the A* agent jumps during its execution (rangesfrom 0 to 100).

Original: Since the environment dynamics are stochastic, for the ground-truth evaluations thecompletion and number of jumps are averaged over 5 episodes (the number of sky tiles does notdepend on the agent’s execution).
Revised: Since the environment dynamics are stochastic, for the ground-truth evaluations thecompletion and number of jumps are averaged over 5 episodes (the number of sky tiles does notdepend on the agent’s execution).

Original:  Experiments
Revised:  Experiments

Original: 6.1 Experiment Design
Revised: 6.1 Experiment Design

Original: Independent variables.
Revised: Independent variables.

Original: In each domain (Maze and Mario), we follow a between-groups design,where the independent variable is the algorithm.
Revised: In each domain (Maze and Mario), we follow a between-groups design,where the independent variable is the algorithm.

Original: We test the following algorithms:
Revised: We test the following algorithms:

Original: DSAGE :
Revised: DSAGE :

Original: The proposed algorithm that includes predicting ancillary agent behavior data and downsam-
Revised: The proposed algorithm that includes predicting ancillary agent behavior data and downsam-

Original: pling the surrogate archive (Sec. 4).
Revised: pling the surrogate archive (Sec. 4).

Original: DSAGE-Only Anc : The proposed algorithm with ancillary data prediction and no downsampling, i.e.,
Revised: DSAGE-Only Anc : The proposed algorithm with ancillary data prediction and no downsampling, i

Original: [Figure]
Revised: e.,selecting all solutions from the surrogate archive. DSAGE-Only Down : The proposed algorithm with downsampling and no ancillary data prediction.

Original: [Figure] version of the proposed algorithm that selects all solutions from thesurrogate archive and does not predict ancillary data.
Revised: DSAGE Basic : The basic version of the proposed algorithm that selects all solutions from the surrogate archive and does not predict ancillary data.

Original: Baseline QD : The QD algorithm without surrogate assistance.
Revised: Baseline QD : The QD algorithm without surrogate assistance.

Original: We follow previous work [16] anduse CMA-ME for the Mario domain.
Revised: We follow previous work [14] anduse CMA-ME for the Mario domain.

Original: Since CMA-ME operates only in continuous spaces, we use MAP-Elites in the discrete Maze domain.
Revised: Since CMA-ME operates only in continuous spaces, we useMAP-Elites in the discrete Maze domain.

Original: Domain Randomization (DR) [86–88] : Algorithm that generates and evaluate random solutions,i.e. wall locations for the maze domain and the latent code to pass through the GAN for the Mariodomain, respectively.

Original: Dependent variables.
Revised: Dependent variables.

Original: We measure the quality and diversity of the solutions with the QD-scoremetric [26](Eq. 1).
Revised: We measure the quality and diversity of the solutions with the QD-scoremetric [24](Eq. 1).

Original: As an additional metric of diversity, we also report the archive coverage.
Revised: As an additional metric of diversity, we also report the archive coverage.

Original: We runeach algorithm for 5 trials in each domain.
Revised: We runeach algorithm for 5 trials in each domain.

Original: Hypothesis.
Revised: Hypothesis.

Original: We hypothesize that DSAGE will result in a better QD-score than DSAGE Basic in alldomains, which in turn will result in better performance than the baseline QD algorithm.
Revised: We hypothesize that DSAGE will result in a better QD-score than DSAGE Basic in alldomains, which in turn will result in better performance than the baseline QD algorithm.

Original: DSAGE, DSAGE Basic, and the baseline QD algorithm will all exceed DR.
Revised: We base this

Original: We base this hypothesis onpreviouswork [25, 17] which shows that QD algorithms outperform random sampling in a variety of domains,as well as previous work [18, 19] which shows that surrogate-assisted MAP-Elites outperformsstandard MAP-Elites in design optimization and Hearthstone domains.
Revised: [Figure] hypothesis on previous work [16, 17] that showed that surrogate-assisted MAP-Elites outperformedstandard MAP-Elites in design optimization and Hearthstone domains.

Original: Furthermore, we expect thatthe additional supervision through ancillary agent behavior data and downsampling will result in
Revised: Furthermore, we expect thatthe additional supervision through ancillary agent behavior data and downsampling will result in

Original: DSAGE performing significantly better than DSAGE Basic.
Revised: DSAGE performing significantly better than DSAGE Basic.

Original: 6.2 Analysis
Revised: 6.2 Analysis

Original: Fig.
Revised: Fig.

Original: 2 summarizes the results obtained by the five algorithms on the Maze and the Mario domains.
Revised: 2 summarizes the results obtained by the five algorithms on the Maze and the Mario domains.

Original: One-way ANOVA tests showed a significant effect of the algorithm on the QD-score for the Maze( F (5 , 24) = 430 .
Revised: One-way ANOVA tests showed a significant effect of the algorithm on the QD-score for the Maze( F (4 , 20) = 126 .

Original: 98 , p < 0 .
Revised: 93 , p < 0 .

Original: and Mario ( F (5 , 24) = 238 .
Revised: and Mario ( F (4 , 20) = 142 .

Original: 09 , p < 0 .
Revised: 09 , p < 0 .

Original: 001 ) domains.
Revised: 001 ) domains.

Original: Post-hoc pairwise comparisons with Bonferroni corrections showed that DSAGE outperformed
Revised: Post-hoc pairwise comparisons with Bonferroni corrections showed that DSAGE outperformed

Original: DSAGE Basic, Baseline QD, and DR in both the Maze and the Mario domains ( p < 0 .001 ).
Revised: DSAGE Basic and Baseline QD in both the Maze and the Mario domains ( p < 0 .

Original: Additionally, DSAGE Basic outperformed MAP-Elitesand DR in the Maze domain ( p < 0 .
Revised: Additionally, DSAGE Basic outperformed MAP-Elites in the Maze domain ( p < 0 .

Original: 001) ,while it performed significantly worse than the QD counterpart, CMA-ME, in the Mario domain ( p =0 . 003) .
Revised: 001) , while it performedsignificantly worse than the QD counterpart, CMA-ME, in the Mario domain ( p = 0 . 003) .

Original: Finally, Baseline QD outperformed DR in both the Maze and Mario domains ( p < 0 . 001 ).

Original: These results show that deep surrogate assisted generation of environments results in significantimprovements compared to quality diversity algorithms without surrogate assistance.
Revised: These results show that deep surrogate assisted generation of environments results in significantimprovements compared to quality diversity algorithms without surrogate assistance.

Original: They also show [Table] [Table] that adding ancillary agent behavior data and downsampling is important in both domains.
Revised: They also showthat adding ancillary agent behavior data and downsampling is important in both domains.

Original: Withoutthese components, DSAGE Basic has limited or no improvement compared to the QD algorithmwithout surrogate assistance.
Revised: Withoutthese components, DSAGE Basic has limited or no improvement compared to the QD algorithmwithout surrogate assistance.

Original: Additionally, domain randomization is significantly worse than DSAGEas well as the baselines.

Original: The archive coverage and consequently the QD score is negligible in the

Original: Mario domain since randomly sampled latent codes led to little diversity in the levels

Original: [Table]

Original: Table 1 shows another metric of the speed-up provided by DSAGE: the number of evaluations (agentsimulations) required to reach a fixed QD score.

Original: We set this fixed QD score to be 10480.8 in the Mazedomain and 1306.11 in the Mario domain, which are the mean QD scores of MAP-Elites and DSAGE

Original: Basic, respectively, in those domains.

Original: DSAGE reaches these QD scores faster than the baselines do.

Original: To assess the quality of the trained surrogate model, we create a combined dataset consisting of datafrom one run of each surrogate assisted algorithm.
Revised: To assess the quality of the trained surrogate model, we create a combined dataset consisting ofdata from one run of each surrogate assisted algorithm.

Original: We use this dataset to evaluate the surrogatemodels trained from separate runs of DSAGE and its variants.
Revised: We use the dataset to evaluate the surrogatemodels trained from separate runs of DSAGE and DSAGE Basic.

Original: Table 2 shows the mean absolute error(MAE) of the predictions by the surrogate models.

Original: The model learned by DSAGE Basic fails topredict the agent-based measures well.
Revised: The model learned by DSAGE Basic fails to predict the agent-based measures well.

Original: It has an MAE of 157.69 for the mean agent path length in Maze and MAE = 10.71 for the number of jumps in Mario.
Revised: It has a mean absolute error (MAE) of 157.69for the mean agent path length in Maze and MAE = 10.71 for the number of jumps in Mario.

Original: In contrast, the model learned by DSAGEmakes more accurate predictions, with MAE = 96.58 for the mean agent path length and MAE =
Revised: Incontrast, the model learned by DSAGE makes more accurate predictions, with MAE = 96.58 for themean agent path length and MAE =

Original: 7.16 for the number of jumps.
Revised: 7.16 for the number of jumps.

Original: We provide the detailed results of the surrogate model predictions in Appendix B.1.
Revised: We explore the contribution ofeach component of DSAGE with an ablation study in Sec. 6.3.

Original: 6.3 Ablation Study
Revised: 6.3 Ablation Study

Original: Sec. 4 describes two key components of DSAGE: (1) self-supervised prediction of ancillary agentbehavior data, and (2) downsampling to select solutions from the surrogate archive.
Revised: Sec. 4 describes two key components of DSAGE: (1) self-supervised prediction of ancillary agentbehavior data, and (2) downsampling to select solutions from the surrogate archive.

Original: We perform anablation study by treating the inclusion of ancillary data prediction (ancillary data / no ancillary data)and the method of selecting solutions from the surrogate archive (downsampling / full selection) asindependent variables.
Revised: We perform anablation study by treating the inclusion of ancillary data prediction (ancillary data / no ancillary data)and the method of selecting solutions from the surrogate archive (downsampling / full selection) as independent variables.

Original: A two-way ANOVA for each domain showed no significant interaction effects.
Revised: A two-way ANOVA for each domain showed no significant interaction effects.

Original: We perform a main effects analysis for each independent variable.
Revised: We perform a main effects analysis for each independent variable.

Original: Inclusion of ancillary data prediction.
Revised: Inclusion of ancillary data prediction.

Original: A main effects analysis for the inclusion of ancillary dataprediction showed that algorithms that predict ancillary agent behavior data (DSAGE, DSAGEOnly Anc) performed significantly better than their counterparts with no ancillary data prediction(DSAGE-Only Down, DSAGE Basic) in both domains ( p < 0 . 001 ).
Revised: A main effects analysis for the inclusion of ancillary dataprediction showed that algorithms that predict ancillary agent behavior data (DSAGE, DSAGEOnly Anc) performed significantly better than their counterparts with no ancillary data prediction(DSAGE-Only Down, DSAGE Basic) in both domains ( p < 0 . 001 ).

Original: Fig.
Revised: Fig.

Original: 2 shows that predicting ancillary agent behavior data also resulted in a larger mean coverage for
Revised: 2 shows that predicting ancillary agent behavior data also results in a larger mean coverage for

Original: Maze, while it has little or no improvement for Mario.
Revised: Maze, while it has little or no improvement for Mario.

Original: Additionally, as shown in Table 2, predictingancillary agent behavior data helped improve the prediction of the mean agent path length in the

Original: Maze domain but provided little improvement to the prediction of the number of jumps in the Mariodomain.

Original: The reason is that in the Maze domain, the mean agent path length is a scaled version of the sum of the agent’s tile occupancy frequency, hence the two-stage process which predicts theoccupancy grid first is essential for improving the accuracy of the model.
Revised: The reason is that in the Maze domain, themean agent path length is a scaled version of the sum of the agent’s tile occupancy frequency, hencethe two-stage process which predicts the occupancy grid first is essential for improving the accuracyof the CNN.

Original: On the other hand, thepresence of a jump in Mario depends not only on cell occupancy, but also on the structure of the leveland the sequence of the occupied cells.
Revised: On the other hand, the presence of a jump in Mario depends not only on cell occupancy,but also on the structure of the level and the sequence of the occupied cells.

Original: Method of selecting solutions from the surrogate archive.
Revised: Method of selecting solutions from the surrogate archive.

Original: A main effects analysis for the methodof selecting solutions from the surrogate archive showed that the algorithms with downsampling(DSAGE, DSAGE-Only Down) performed significantly better than their counterparts with no downsampling (DSAGE-Only Anc, DSAGE-Basic) in both domains ( p < 0 . 001 ).
Revised: A main effects analysis for the methodof selecting solutions from the surrogate archive showed that the algorithms with downsampling(DSAGE, DSAGE-Only Down) performed significantly better than their counterparts with no downsampling (DSAGE-Only Anc, DSAGE-Basic) in both domains ( p < 0 . 001 ).

Original: A major advantage of downsampling is that it decreases the number of ground-truth evaluations ineach outer iteration.
Revised: A major advantage of downsampling is that it decreases the number of ground-truth evaluations ineach outer iteration.

Original: Thus, for a fixed evaluation budget, downsampling results in a greater numberof outer iterations.
Revised: Thus, for a fixed evaluation budget, downsampling results in a greater numberof outer iterations.

Original: For instance, in the Maze domain, runs without downsampling had only 6-7outer iterations, while runs with downsampling had approximately 220 outer iterations.
Revised: For instance, in the Maze domain, runs without downsampling had only 6-7outer iterations, while runs with downsampling had approximately 220 outer iterations.

Original: More outeriterations leads to more training and thus higher accuracy of the surrogate model.
Revised: More outeriterations lead to more training and thus higher accuracy of the surrogate model.

Original: In turn, a moreaccurate surrogate model will generate a better surrogate archive in the inner loop.
Revised: In turn, a moreaccurate surrogate model will generate a better surrogate archive in the inner loop.

Original: The second advantage of downsampling is that it selects solutions evenly from all regions of themeasure space, thus creating a more balanced dataset.
Revised: The second advantage of downsampling is that it selects solutions evenly from all regions of themeasure space, thus creating a more balanced dataset.

Original: This helps train the surrogate model in parts ofthe measure space that are not frequently visited.
Revised: This helps train the surrogate model in parts ofthe measure space that are not frequently visited.

Original: We include an additional baseline in Appendix Din which we selecta subset of solutions uniformly at random from the surrogate archive instead ofdownsampling.
Revised: We include an additional baseline in Appendix D inwhich we select solutions uniformly at random from the surrogate archive instead of downsampling.

Original: We observe that downsampling has a slight advantage over uniform random samplingin the Maze domain.

Original: Furthermore, if instead of downsampling we sampled multiple solutions from nearby regions of thesurrogate archive, the prediction errors could cause the solutions to collapse to a single cell in theground-truth archive, resulting in many solutions being discarded.
Revised: Furthermore, if instead of downsampling we sampled multiple solutions from nearby regions of thesurrogate archive, the prediction errors could cause the solutions to collapse to a single cell in theground-truth archive, resulting in many solutions being discarded.

Original: Overall, our ablation study shows that both predicting the occupancy grid as ancillary data anddownsampling the surrogate archive independently help improve the performance of DSAGE.
Revised: Overall, our ablation study shows that both predicting the occupancy grid as ancillary data anddownsampling the surrogate archive independently help improve the performance of DSAGE.

Original: 6.4 Qualitative Results
Revised: 6.4 Qualitative Results

Original: Fig.
Revised: Fig.

Original: 3 and Fig.
Revised: 3 and Fig.

Original: 4 show example environments generated by DSAGE in the Maze and the Mariodomains.
Revised: 4 show example environments generated by DSAGE in the Maze and the Mariodomains.

Original: Having the mean agent path length as a measure in the Maze domain results in environments ofvarying difficulty for the ACCEL agent.
Revised: Having the mean agent path length as a measure in the Maze domain results in environments ofvarying difficulty for the ACCEL agent.

Original: For instance, we observe that the environment in Fig.
Revised: For instance, we observe that the environment in Fig.

Original: 3(a)has very few walls, yet the ACCEL agent gets stuck in the top half of the maze and is unable to findthe goal within the allotted time.
Revised: 3(a)has very few walls, yet the ACCEL agent gets stuck in the top half of the maze and is unable to findthe goal within the allotted time.

Original: On the other hand, the environment in Fig. 3(d) is cluttered andthere are multiple dead-ends, yet the ACCEL agent is able to reach the goal.
Revised: On the other hand, the environment in Fig. 3(d) is cluttered andthere are multiple dead-ends, yet the ACCEL agent is able to reach the goal.

Original: Fig.
Revised: Fig.

Original: 4 shows that the generated environments result in qualitatively diverse behaviors for the Marioagent too.
Revised: 4 shows that the generated environments result in qualitatively diverse behaviors for the Marioagent too.

Original: Level (b) only has a few sky tiles and is mostly flat, resulting in a small number of jumps.
Revised: Level (b) only has a few sky tiles and is mostly flat, resulting in a small number of jumps.

Original: Level (c) has a “staircase trap” on the right side, forcing the agent to perform continuous jumps toescape and complete the level.
Revised: Level (c) has a “staircase trap” on the right side, forcing the agent to perform continuous jumps toescape and complete the level.

Original: We include videos of the playthroughs in the supplemental material.
Revised: We include videos of the playthroughs in the supplemental material.

Original:  Societal Impacts
Revised:  Societal Impacts

Original: By introducing surrogate models into quality diversity algorithms, we can efficiently generateenvironments that result in diverse agent behaviors.
Revised: By introducing surrogate models into quality diversity algorithms, we can efficiently generateenvironments that result in diverse agent behaviors.

Original: While we focused on an RL agent in a Mazedomain and a planning agent in a Mario game domain, our method can be applied to a variety ofagents and domains.
Revised: While we focused on an RL agent in a Mazedomain and a planning agent in a Mario game domain, our method can be applied to a variety ofagents and domains.

Original: This can help with testing the robustness of agents, attaining insights about theirbehavior, and discovering edge cases before real-world deployment.
Revised: This can help with testing the robustness of agents, attaining insights about theirbehavior, and discovering edge cases before real-world deployment.

Original: Furthermore, we anticipate thatin the future, closing the loop between environment generation and agent training can improve theability of agents to generalize to new settings and thus increase their widespread use.
Revised: Furthermore, we anticipate that [Figure] [Figure] closing the loop between environment generation and agent training can improve the ability of agentsto generalize to new settings and thus increase their widespread use.

Original: Our work may also have negative impacts.
Revised: Our work may also have negative impacts.

Original: Training agents in diverse environments can be consideredas a step towards open-ended evolution [89], which raises concerns about the predictability and safetyof the emergent agent behaviors [90, 91].
Revised: Training agents in diverse environments can be consideredas a step towards open-ended evolution [83], which raises concerns about the predictability and safetyof the emergent agent behaviors [84, 85].

Original: Discovering corner cases that result in unwanted behaviorsor catastrophic failures may also be used maliciously to reveal vulnerabilities in deployed agents [92]. [Figure] [Figure]
Revised: Discovering corner cases that result in unwanted behaviorsor catastrophic failures may also be used maliciously to reveal vulnerabilities in deployed agents [86].

Original:  Limitations and Future Work
Revised:  Limitations and Future Work

Original: Automatic environment generation is a rapidly growing research area with a wide range of applications,including designing video game levels [46, 53, 54], training and testing autonomous agents [74, 76,1, 3, 4] [Figure] and discovering failures in human-robot interaction [17, 16].
Revised: Automatic environment generation is a rapidly growing research area with a wide range of applications,including designing video game levels [45, 52, 53], training and testing autonomous agents [73, 75,1, 3, 4], and discovering failures in human-robot interaction [15, 14].

Original: We introduce the DSAGEalgorithm, which efficiently generates a diverse collection of environments via deep surrogate modelsof agent behavior [Figure]
Revised: We introduce the DSAGEalgorithm, which efficiently generates a diverse collection of environments via deep surrogate modelsof agent behavior.

Original: Our paper has several limitations.
Revised: Our paper has several limitations.

Original: First, occupancy grid prediction does not encode temporalinformation about the agent.
Revised: First, occupancy grid prediction does not encode temporalinformation about the agent.

Original: On one hand, this prediction allows us to avoid the compounding errorproblem of model-based RL [93].
Revised: On one hand, this prediction allows us to avoid the compounding errorproblem of model-based RL [87].

Original: On the other hand, forgoing temporal information makes it harderto predict some behaviors, such as the number of jumps in Mario.
Revised: On the other hand, forgoing temporal information makes it harderto predict some behaviors, such as the number of jumps in Mario.

Original: We will explore this trade-off infuture work.
Revised: We will explore this trade-off infuture work.

Original: Furthermore, we have studied 2D domains where a single ground-truth evaluation lasts between afew seconds and a few minutes.
Revised: Furthermore, we have studied 2D domains where a single ground-truth evaluation lasts between afew seconds and a few minutes.

Original: We are excited about the use of surrogate models to predict theperformance of agents in more complex domains with expensive, high-fidelity simulations [94]
Revised: We are excited about the use of surrogate models to predict theperformance of agents in more complex domains with expensive, high-fidelity simulations [88].

Original: Checklist
Revised: Checklist

Original: For all authors...
Revised: For all authors...

Original: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’scontributions and scope?
Revised: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s

Revised: contributions and scope?

Original: [Yes] We claim that our algorithm significantly outperforms
Revised: [Yes] We claim that our algorithm significantly outperforms

Original: QD environment generation algorithms.
Revised: QD environment generation algorithms.

Original: The claim is supported by results and thestatistical tests shown in Sec. 6.2.
Revised: The claim is supported by results and the statistical tests shown in Sec. 6.2.

Original: (b) Did you describe the limitations of your work?
Revised: (b) Did you describe the limitations of your work?

Original: [Yes] We list the limitations of our
Revised: [Yes] We list the limitations of our

Original: work in Sec. 8.
Revised: work in Sec. 8.

Original: (c) Did you discuss any potential negative societal impacts of your work?
Revised: (c) Did you discuss any potential negative societal impacts of your work?

Original: [Yes] We discuss
Revised: [Yes] We discuss

Original: the potential positive and negative impacts of this work in Sec.
Revised: the potential positive and negative impacts of this work in Sec.

Original: (d) Have you read the ethics review guidelines and ensured that your paper conforms to
Revised: (d) Have you read the ethics review guidelines and ensured that your paper conforms to

Original: them?
Revised: them?

Original: [Yes]
Revised: [Yes]

Original: If you are including theoretical results...
Revised: If you are including theoretical results...

Original: (a) Did you state the full set of assumptions of all theoretical results?
Revised: (a) Did you state the full set of assumptions of all theoretical results?

Original: [N/A]
Revised: [N/A]

Original: (b) Did you include complete proofs of all theoretical results?
Revised: (b) Did you include complete proofs of all theoretical results?

Original: [N/A]
Revised: [N/A]

Original: If you ran experiments...
Revised: If you ran experiments...

Original: (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
Revised: (a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL

Original: [Yes] We provide thecode in the supplemental material for running all the experiments specified in Sec. 6.
Revised: [Yes] We provide thecode in the supplemental material for running all the experiments specified in Sec. 6.

Original: The code contains a README file with the instructions to reproduce the experiments.
Revised: The code contains a README file with the instructions to reproduce the experiments.

Original: (b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?
Revised: (b) Did you specify all the training details (e g., data splits, hyperparameters, how they were chosen)?

Original: [Yes] The main details are presented in Sec. 5 and Sec. 6.
Revised: [Yes] The main details are presented in Sec. 5 and Sec. 6.

Original: Other details
Revised: Other details

Original: are provided in Appendix C and also available in the attached code.
Revised: are provided in Appendix C and also available in the attached code

Original: (c) Did you report error bars (e.g., with respect to the random seed after running exper-
Revised: (c) Did you report error bars (e.g., with respect to the random seed after running exper-

Original: iments multiple times)?
Revised: iments multiple times)?

Original: [Yes] All results shown in Sec.
Revised: [Yes] All results shown in Sec.

Original: 6.2 and Sec.
Revised: 6.2 and Sec.

Original: 6.3 have the corresponding standard error and the inferences are supported by relevant statisticaltests.
Revised: 6.3 have thecorresponding standard error and the inferences are supported by relevant statistical tests.

Original: (d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
Revised: (d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?

Original: [Yes] We include the description of the
Revised: [Yes] We include the description of the

Original: computational resources used for the runs in Appendix C.
Revised: computational resources used for the runs in Appendix C.

Original: If you are using existing assets (e
Revised: If you are using existing assets (e

Original: g., code, data, models) or curating/releasing new assets...
Revised: g., code, data, models) or curating/releasing new assets...

Original: (a) If your work uses existing assets, did you cite the creators?
Revised: (a) If your work uses existing assets, did you cite the creators?

Original: [Yes] We have used existingopen source code for the Mario simulator and the A* agent, the Maze environment,and QD algorithm implementation.
Revised: [Yes] We have used existing open source code for the Mario simulator and the A* agent, the Maze environment, and QD algorithm implementation.

Original: We have cited the creators either in Sec.
Revised: We have cited the creators either in Sec.

Original: 5 or inAppendix A. We obtained ACCEL agents directly from the original authors with their
Revised: 5 or in Appendix A. We obtained ACCEL agents directly from the original authors with their

Original: consent.
Revised: consent.

Original: We have cited the paper which introduced the algorithm and will acknowledgethe authors for providing the pre-trained agents.
Revised: We have cited the paper which introduced the algorithm and will acknowledge the authors for providing the pre-trained agents.

Original: (b) Did you mention the license of the assets?
Revised: (b) Did you mention the license of the assets?

Original: [Yes] The licenses to the external code andassets are added to the README file in the code that we provide with the supplemental
Revised: [Yes] The licenses to the external code and assets are added to the README file in the code that we provide with the supplemental

Original: material.
Revised: material.

Original: (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [Yes]
Revised: [Yes]

Original: [Table]

Original: We include videos of the agent’s playthrough in the supplemental material.
Revised: We include videos of the agent’s playthrough in the supplemental material.

Original: (d) Did you discuss whether and how consent was obtained from people whose data you’re[Table]
Revised: (d) Did you discuss whether and how consent was obtained from people whose data you’re

Original: using/curating?
Revised: using/curating?

Original: [N/A]
Revised: [N/A]

Original: (e) Did you discuss whether the data you are using/curating contains personally identifiableinformation or offensive content?
Revised: (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?

Original: [N/A]
Revised: [N/A]

Original: If you used crowdsourcing or conducted research with human subjects...
Revised: If you used crowdsourcing or conducted research with human subjects...

Original: (a) Did you include the full text of instructions given to participants and screenshots, if
Revised: (a) Did you include the full text of instructions given to participants and screenshots, if

Original: applicable?
Revised: applicable?

Original: [N/A]
Revised: [N/A]

Original: [Table] (b) Did you describe any potential participant risks, with links to Institutional Review[Table]
Revised: (b) Did you describe any potential participant risks, with links to Institutional Review

Original: Board (IRB) approvals, if applicable?
Revised: Board (IRB) approvals, if applicable?

Original: [N/A]
Revised: [N/A]

Original: [Table]

Original: (c) Did you include the estimated hourly wage paid to participants and the totalamount
Revised: (c) Did you include the estimated hourly wage paid to participants and the total

Original: [Table]
Revised: amountspent

Original: spent on participant compensation?
Revised: on participant compensation?

Original: [N/A]
Revised: [N/A

Original: [Table]