Original: Deep Surrogate Assisted Generation of Environments
Revised: Deep Surrogate Assisted Generation of Environments

Original: Anonymous Author(s)

Original: AffiliationAddress email

Original: Abstract
Revised: Abstract

Original: Recent progress in reinforcement learning (RL) has started producing generallycapable agents that can solve a distribution of complex environments.
Revised: Recent progress in reinforcement learning (RL) has started producing generally capable agents that can solve a distribution of complex environments.

Original: These agentsare typically tested on fixed, human-authored environments.
Revised: These agents are typically tested on fixed, human-authored environments.

Original: On the other hand,quality diversity (QD) optimization has been proven to be an effective component ofenvironment generation algorithms, which can generate collections of high-qualityenvironments that are diverse in the resulting agent behaviors.
Revised: On the other hand, quality diversity (QD) optimization has been proven to be an effective component of environment generation algorithms, which can generate collections of high-quality environments that are diverse in the resulting agent behaviors.

Original: However, thesealgorithms require potentially expensive simulations of agents on newly generatedenvironments.
Revised: However, these algorithms require potentially expensive simulations of agents on newly generated environments.

Original: We propose Deep Surrogate Assisted Generation of Environments(DSAGE), a sample-efficient QD environment generation algorithm that maintainsa deep surrogate model for predicting agent behaviors in new environments.
Revised: We propose Deep Surrogate Assisted Generation of Environments (DSAGE), a sample-efficient QD environment generation algorithm that maintains a deep surrogate model for predicting agent behaviors in new environments.

Original: Resultsin two benchmark domains show that DSAGE significantly outperforms existing
Revised: Results in two benchmark domains show that DSAGE significantly outperforms existing

Original: QD environment generation algorithms in discovering collections of environmentsthat elicit diverse behaviors of a state-of-the-art RL agent and a planning agent.
Revised: QD environment generation algorithms in discovering collections of environments that elicit diverse behaviors of a state-of-the-art RL agent and a planning agent.

Revised: Our source code and videos are available at https://dsagepaper.github.io/

Original:  Introduction
Revised:  Introduction

Original: We present an efficient method of automatically generating a collection of environments that elicitdiverse agent behaviors.
Revised: We present an efficient method of automatically generating a collection of environments that elicit diverse agent behaviors.

Original: As a motivating example, consider deploying a robot agent at scale in avariety of home environments.
Revised: As a motivating example consider deploying a robot agent at scale in a variety of home environments.

Original: The robot should generalize by performing robustly not only in testhomes, but in any end user’s home.
Revised: The robot should generalize by performing robustly not only in test homes but in any end user’s home.

Original: To enable such generalization, the test environments shouldhave good coverage for the robot agent.
Revised: To validate agent generalization, the test environments should have good coverage for the robot agent.

Original: However, obtaining such coverage may be difficult, as thegenerated environments would depend on the application domain, e.g. kitchen or living room, and onthe specific agent we want to test, since different agents exhibit different behaviors.
Revised: However, obtaining such coverage may be difficult as the generated environments would depend on the application domain, e.g. kitchen or living room and on the specific agent we want to test, since different agents exhibit different behaviors.

Original: To enable generalization of autonomous agents to new environments with differing levels of complexity, previous work on open-ended learning [1, 2] has integrated the environment generation and theagent training processes.
Revised: To enable generalization of autonomous agents to new environments with differing levels of complexity, previous work on open-ended learning [1, 2] has integrated the environment generation and the agent training processes.

Original: The interplay between the two processes acts as a natural curriculum for theagents to learn robust skills that generalize to new, unseen environments [3–5].
Revised: The interplay between the two processes acts as a natural curriculum for the agents to learn robust skills that generalize to new, unseen environments [3–5].

Original: The performance ofthese agents has been evaluated either in environments from the training distribution [1, 2, 5] or insuites of manually authored environments [3, 6, 4].
Revised: The performance of these agents has been evaluated either in environments from the training distribution [1, 2, 5] or in suites of manually authored environments [3, 6, 4].

Original: As a step towards testing generalizable agents, there has been increasing interest in competitions [7, 8]that require agents to generalize to new game layouts.
Revised: As a step towards testing generalizable agents, there has been increasing interest in competitions [7, 8] that require agents to generalize to new game layouts.

Original: Despite the recent progress of deep learningagents in fixed game domains, e.g. in Chess [9], Go [10], Starcraft [11], and Poker [12, 13], it hasbeen rule-based agents that have succeeded in these competitions [8]. Such competitions also rely onmanually authored game levels as a test set, handcrafted by a human designer. While manually authored environments are important for standardized testing, creating these environments can be tedious and time-consuming. Additionally, manually authored test suites are ofteninsufficient for eliciting the diverse range of possible agent behaviors.
Revised: Despite the recent progress of deep learning

Original: Instead, we would like aninteractive test set that proposes an environment, observes the agent’s performance and behavior, and

Original: Sample Solution
Revised: Sample Solution

Original: UpdateSurrogateArchive
Revised: UpdateSurrogateArchive

Original: Extract Predicted Agent Performance and
Revised: Extract Predicted Agent Performance and

Original: Measured Behavior
Revised: Measured Behavior

Original: Generate Environments from Solutions and Evaluate
Revised: Generate Environments from Solutions and Evaluate

Original: Extract Ground Truth Agent Performance and Behavior Data
Revised: Extract Ground Truth Agent Performance and Behavior Data

Original: Downsample Archive
Revised: Downsample Archive

Original: Update Dataset
Revised: Update Dataset

Original: Update Ground Truth Archive
Revised: Update Ground Truth Archive

Original: Generate Environment
Revised: Generate Environment

Original: Train the Surrogate Model on the Dataset
Revised: Train the Surrogate Model on the Dataset

Original: Environment Generation
Revised: Environment Generation

Original: Occupancy Prediction
Revised: Occupancy Prediction

Original: Stack with Occupancy Grid
Revised: Stack with Occupancy Grid

Original: Self-supervised Agent Occupancy Prediction
Revised: Self-supervised Agent Occupancy Prediction

Original: A A
Revised: A A

Original: Model Exploitation
Revised: Model Exploitation

Original: Agent Simulation
Revised: Agent Simulation

Revised: Model Improvement agents in fixed game domains, e.g. in Chess [9], Go [10], Starcraft [11], and Poker [12, 13], it has been rule-based agents that have succeeded in these competitions [8].

Original: Model Improvement then proposes new environments that diversify the agent behaviors, based on what the system haslearned from previous execution traces of the agent.
Revised: Such competitions also rely on manually authored game levels as a test set, handcrafted by a human designer. While manually authored environments are important for standardized testing, creating these environments can be tedious and time-consuming. Additionally, manually authored test suites are often insufficient for eliciting the diverse range of possible agent behaviors. Instead, we would like an interactive test set that proposes an environment, observes the agent’s performance and behavior, and then proposes new environments that diversify the agent behaviors, based on what the system has learned from previous execution traces of the agent.

Original: To address collecting environments with diverse agent behaviors, prior work frames the problemas a quality diversity (QD) problem [15–17].
Revised: To address collecting environments with diverse agent behaviors, prior work frames the problem as a quality diversity (QD) problem [15–17].

Original: A QD problem consists of an objective function, e.g.whether the agent can solve the environment, and measure functions, e.g. how long the agent takesto complete their task.
Revised: A QD problem consists of an objective function, e.g. whether the agent can solve the environment, and measure functions, e.g. how long the agent takes to complete their task.

Original: The measure functions quantify the behavior we would like to vary in theagent, allowing practitioners to specify the case coverage they would like to see in the domain theyare testing.
Revised: The measure functions quantify the behavior we would like to vary in the agent, allowing practitioners to specify the case coverage they would like to see in the domain they are testing.

Original: While QD algorithms can generate diverse collections of environments, they require alarge number of environment evaluations to produce the collection, and each of these evaluationsrequires multiple time-consuming simulated executions of potentially stochastic agent policies.
Revised: While QD algorithms can generate diverse collections of environments, they require a large number of environment evaluations to produce the collection, and each of these evaluations requires multiple time-consuming simulated executions of potentially stochastic agent policies.

Original: We study how deep surrogate models that predict agent performance can accelerate the generationof environments that are diverse in agent behaviors .
Revised: We study how deep surrogate models that predict agent performance can accelerate the generation of environments that are diverse in agent behaviors .

Original: We draw upon insights from model-basedquality diversity algorithms that have been previously shown to improve sample efficiency in designoptimization [18] and Hearthstone deckbuilding [19].
Revised: We draw upon insights from model-based quality diversity algorithms that have been previously shown to improve sample efficiency in design optimization [18] and Hearthstone deckbuilding [19].

Original: Environments present a much more complexprediction task because the evaluation of environments involves simulating stochastic agent policies,and small changes in the environment may result in large changes in the emergent agent behaviors [20].
Revised: Environments present a much more complex prediction task because the evaluation of environments involves simulating stochastic agent policies, and small changes in the environment may result in large changes in the emergent agent behaviors [20].

Original: We make the following contributions: (1) We propose the use of deep surrogate models to predictagent performance in new environments.
Revised: We make the following contributions: (1) We propose the use of deep surrogate models to predict agent performance in new environments.

Original: Our algorithm, Deep Surrogate Assisted Generation of
Revised: Our algorithm, Deep Surrogate Assisted Generation of

Original: Environments (DSAGE) (Fig. 1), integrates deep surrogate models into quality diversity optimizationto efficiently generate diverse environments.
Revised: Environments (DSAGE) (Fig. 1), integrates deep surrogate models into quality diversity optimization to efficiently generate diverse environments.

Original: (2) We show in two benchmark domains from previous work, a Maze domain [3, 4] with a trained ACCEL agent [4] and a Mario domain [21, 16] with an
Revised: (2) We show in two benchmark domains from previous work, a Maze domain [3, 4] with a trained ACCEL agent [4] and a Mario domain [21, 16] with an

Original: A* agent [22], that DSAGE outperforms state-of-the-art QD algorithms in discovering diverse agentbehaviors.
Revised: A* agent [22], that DSAGE outperforms state-of-the-art QD algorithms in discovering diverse agent behaviors.

Original: (3) We show with ablation studies that training the surrogate model with ancillary agentbehavior data and downsampling a subset of solutions from the surrogate archive results in substantialimprovements in performance, compared to the surrogate models of previous work [19].
Revised: (3) We show with ablation studies that training the surrogate model with ancillary agent behavior data and downsampling a subset of solutions from the surrogate archive results in substantial improvements in performance, compared to the surrogate models of previous work [19].

Original:  Problem Definition
Revised:  Problem Definition

Original: Quality diversity (QD) optimization.
Revised: Quality diversity (QD) optimization.

Original: We adopt the QD problem definition from previous work [23].
Revised: We adopt the QD problem definition from previous work [23].

Original: A QD optimization problem specifies an objective function f : R n → R and a joint measure functionm : R n → R m .
Revised: A QD optimization problem specifies an objective function f : R n → R and a joint measure function m : R n → R m .

Original: For each element s ∈ S , where S ⊆
Revised: For each element s ∈ S , where S ⊆

Original: R m is the range of the measure function, the
Revised: R m is the range of the measure function, the

Original: QD goal is to find a solution θ ∈ R n such that m ( θ )
Revised: QD goal is to find a solution θ ∈ R n such that m ( θ )

Original: =
Revised: =

Original: s and f ( θ ) is maximized.
Revised: s and f ( θ ) is maximized.

Original: Since the range of the measure function can be continuous, we restrict ourselves to algorithms fromthe MAP-Elites family [24, 25] that discretize this space into a finite number of M cells.
Revised: Since the range of the measure function can be continuous, we restrict ourselves to algorithms from the MAP-Elites family [24, 25] that discretize this space into a finite number of M cells.

Original: A solutionθ is mapped to a cell based on its measure m ( θ ) .
Revised: A solution θ is mapped to a cell based on its measure m ( θ ) .

Original: The solutions that occupy cells form an archive ofsolutions.
Revised: The solutions that occupy cells form an archive of solutions.

Original: Our goal is to find solutions θ i , i ∈ { 1 , ..., M } that maximize the objective f for all cellsin the measure space.
Revised: Our goal is to find solutions θ i , i ∈ { 1 , ..., M } that maximize the objective f for all cells in the measure space.

Original: [Equation]
Revised: [Equation]

Original: The computed sum in Eq.
Revised: The computed sum in Eq.

Original: 1 is defined as the QD-score [26], where empty cells have an objectivevalue of 0.
Revised: 1 is defined as the QD-score [26], where empty cells have an objective value of 0.

Original: A second metric of the performance of a QD algorithm is coverage of the measure space,defined as the proportion of cells that are filled in by solutions: 1 M (cid:80) Mi
Revised: A second metric of the performance of a QD algorithm is coverage of the measure space, defined as the proportion of cells that are filled in by solutions: 1 M (cid:80) Mi

Original: =1 1 θ i .
Revised: =1 1 θ i .

Original: QD for environment generation.
Revised: QD for environment generation.

Original: We assume a single agent acting in an environment parameterizedby θ ∈ R n .
Revised: We assume a single agent acting in an environment parameterized by θ ∈ R n .

Original: The environment parameters can be locations of different objects or latent variables thatare passed as inputs to a generative model [27].
Revised: The environment parameters can be locations of different objects or latent variables that are passed as inputs to a generative model [27].

Original: 1 A QD algorithm generates new solutions θ andevaluates them by simulating the agent on the environment parameterized by θ .
Revised: 2 A QD algorithm generates new solutions θ and evaluates them by simulating the agent on the environment parameterized by θ .

Original: The evaluation returnsan objective value f and measure values m .
Revised: The evaluation returns an objective value f and measure values m .

Original: The QD algorithm attempts to generate environmentsthat maximize f but are diverse with respect to the measures m .
Revised: The QD algorithm attempts to generate environments that maximize f but are diverse with respect to the measures m .

Original:  Background and Related Work
Revised:  Background and Related Work

Original: Quality diversity (QD) optimization.
Revised: Quality diversity (QD) optimization.

Original: QD optimization originated in the genetic algorithm community with diversity optimization [28], the predecessor to QD.
Revised: QD optimization originated in the genetic algorithm community with diversity optimization [28], the predecessor to QD.

Original: Later work introduced objectives todiversity optimization and resulted in the first QD algorithms: Novelty Search with Local Competition [29] and MAP-Elites [25, 24].
Revised: Later work introduced objectives to diversity optimization and resulted in the first QD algorithms: Novelty Search with Local Competition [29] and MAP-Elites [25, 24].

Original: The QD community has grown beyond its genetic algorithmroots, with algorithms being proposed based on gradient ascent [23], Bayesian optimization [30],differential evolution [31], and evolution strategies [14, 32, 33].
Revised: The QD community has grown beyond its genetic algorithm roots, with algorithms being proposed based on gradient ascent [23], Bayesian optimization [30], differential evolution [31], and evolution strategies [14, 32, 33].

Original: QD algorithms have applications indamage recovery in robotics [24], reinforcement learning [34–36], and generative design [18, 37].
Revised: QD algorithms have applications in damage recovery in robotics [24], reinforcement learning [34–36], and generative design [18, 37].

Original: Among the QD algorithms, those of particular interest to us are the model-based ones.
Revised: Among the QD algorithms, those of particular interest to us are the model-based ones.

Original: Currentmodel-based [38, 39] QD algorithms either (1) learn a surrogate model of the objective and measurefunctions [18, 40, 41], e.g. a Gaussian process or neural network, (2) learn a generative modelof the representation parameters [42, 43], or (3) draw inspiration from model-based RL [44, 45].
Revised: Current model-based [38, 39] QD algorithms either (1) learn a surrogate model of the objective and measure functions [18, 40, 41], e.g. a Gaussian process or neural network, (2) learn a generative model of the representation parameters [42, 43], or (3) draw inspiration from model-based RL [44, 45].

Original: In particular, Deep Surrogate Assisted MAP-Elites (DSA-ME)
Revised: In particular, Deep Surrogate Assisted MAP-Elites (DSA-ME)

Original: [19] trains a deep surrogate modelon a diverse dataset of solutions generated by MAP-Elites and then leverages the model to guide
Revised: [19] trains a deep surrogate model on a diverse dataset of solutions generated by MAP-Elites and then leverages the model to guide

Original: MAP-Elites.
Revised: MAP-Elites.

Original: However, DSA-ME has only been applied to Hearthstone deck building, a simplerprediction problem than predicting agent behavior in generated environments.
Revised: However, DSA-ME has only been applied to Hearthstone deck building, a simpler prediction problem than predicting agent behavior in generated environments.

Original: Additionally, DSAME is specific to MAP-Elites only and cannot run other QD algorithms to exploit the surrogatemodel.
Revised: Additionally, DSA-ME is specific to MAP-Elites only and cannot run other QD algorithms to exploit the surrogate model.

Original: Furthermore, DSA-ME is restricted to direct search and cannot integrate generative models togenerate environments that match a provided dataset.
Revised: Furthermore, DSA-ME is restricted to direct search and cannot integrate generative models to generate environments that match a provided dataset.

Original: Automatic environment generation.
Revised: Automatic environment generation.

Original: Automatic environment generation algorithms have beenproposed in a variety of fields.
Revised: Automatic environment generation algorithms have been proposed in a variety of fields.

Original: Methods between multiple communities often share generation techniques, but differ in how each community applies the generation algorithms.
Revised: Methods between multiple communities often share generation techniques, but differ in how each community applies the generation algorithms.

Original: For example, in the procedural content generation (PCG) field [46], an environment generatorproduces video game levels that result in player enjoyment.
Revised: For example, in the procedural content generation (PCG) field [46], an environment generator produces video game levels that result in player enjoyment.

Original: Since diversity of player experienceand game mechanics is valued in games, many level generation systems incorporate QD optimization [47, 16, 48–52].
Revised: Since diversity of player experience and game mechanics is valued in games, many level generation systems incorporate QD optimization [47, 16, 48–52].

Original: The procedural content generation via machine learning (PCGML) [53, 54]subfield studies environment generators that incorporate machine learning techniques such as Markov
Revised: The procedural content generation via machine learning (PCGML) [53, 54] subfield studies environment generators that incorporate machine learning techniques such as Markov

Original: Chains [55], probabilistic graphical models [56], LSTMs [57], generative models [58–61], andreinforcement learning [62, 63].
Revised: Chains [55], probabilistic graphical models [56], LSTMs [57], generative models [58–61], and reinforcement learning [62, 63].

Original: Prior work [64] has leveraged surrogate models trained on offlinedata to accelerate search-based PCG [65].
Revised: Prior work [64] has leveraged surrogate models trained on offline data to accelerate search-based PCG [65].

Original: Environment generation methods have also been proposed by the scenario generation community inrobotics.
Revised: Environment generation methods have also been proposed by the scenario generation community in robotics.

Original: Early work explored automated methods for generating road layouts, vehicle arrangements,and vehicle behaviors for testing autonomous vehicles [66–72].
Revised: Early work explored automated methods for generating road layouts, vehicle arrangements, and vehicle behaviors for testing autonomous vehicles [66–72].

Original: Outside of autonomous vehicles,prior work [73] evaluates robot motion planning algorithms by generating environments that targetspecific motion planning behaviors.
Revised: Outside of autonomous vehicles, prior work [73] evaluates robot motion planning algorithms by generating environments that target specific motion planning behaviors.

Original: In human-robot interaction, QD algorithms have been applied asenvironment generators to find failures in shared autonomy systems [17] and human-aware plannerstested in the collaborative Overcooked domain [15]
Revised: In human-robot interaction, QD algorithms have been applied as environment generators to find failures in shared autonomy systems [17] and human-aware planners tested in the collaborative Overcooked domain [15]

Original: Environment generation can also help improve the generality of RL agents.
Revised: Environment generation can also help improve the generality of RL agents.

Original: Prior work proposesdirectly applying PCG level generation algorithms to improve the robustness of RL [74, 75] or tobenchmark RL agents [76].
Revised: Prior work proposes directly applying PCG level generation algorithms to improve the robustness of RL [74, 75] or to benchmark RL agents [76].

Original: Paired Open-ended Trailblazer (POET)
Revised: Paired Open-ended Trailblazer (POET)

Original: [1, 2] coevolves a population ofboth agents and environments to discover specialized agents that solve complex tasks.
Revised: [1, 2] coevolves a population of both agents and environments to discover specialized agents that solve complex tasks.

Original: POET inspireda variety of open-ended coevolution algorithms [77–79, 5].
Revised: POET inspired a variety of open-ended coevolution algorithms [77–79, 5].

Original: Later work proposes the PAIRED [3],
Revised: Later work proposes the PAIRED [3],

Original: PLR [80, 6], and ACCEL [4] algorithms that train a single generally capable agent by maximizingthe regret between a pair of agents.
Revised: PLR [80, 6], and ACCEL [4] algorithms that train a single generally capable agent by maximizing the regret between a pair of agents.

Original: These methods generate environments in parallel with anagent to create an automatic training curriculum.
Revised: These methods generate environments in parallel with an agent to create an automatic training curriculum.

Original: However, the authors validate these methods onhuman-designed environments [81].
Revised: However, the authors validate these methods on human-designed environments [81].

Original: Our work proposes a method that automatically generates validenvironments that reveal diverse behaviors of these more general RL agents.
Revised: Our work proposes a method that automatically generates valid environments that reveal diverse behaviors of these more general RL agents.

Original:  Deep Surrogate Assisted Generation of Environments (DSAGE)
Revised:  Deep Surrogate Assisted Generation of Environments (DSAGE)

Original: Algorithm.
Revised: Algorithm.

Original: We propose the Deep Surrogate Assisted Generation of Environments (DSAGE) algorithm for discovering environments that elicit diverse agent behaviors.
Revised: We propose the Deep Surrogate Assisted Generation of Environments (DSAGE) algorithm for discovering environments that elicit diverse agent behaviors.

Original: Akin to the MAP-Elites familyof QD algorithms, DSAGE maintains a ground-truth archive where solutions are stored based ontheir ground-truth evaluations.
Revised: Akin to the MAP-Elites family of QD algorithms, DSAGE maintains a ground-truth archive where solutions are stored based on their ground-truth evaluations.

Original: Simultaneously, DSAGE also trains and exploits a deep surrogatemodel for predicting the behavior of a fixed agent in new environments.
Revised: Simultaneously, DSAGE also trains and exploits a deep surrogate model for predicting the behavior of a fixed agent in new environments.

Original: The QD optimization occursin three phases that take place in an outer loop: model exploitation, agent simulation, and modelimprovement (Fig. 1).
Revised: The QD optimization occurs in three phases that take place in an outer loop: model exploitation, agent simulation, and model improvement (Fig. 1).

Original: Algorithm 1 provides the pseudocode for the DSAGE algorithm.
Revised: Algorithm 1 provides the pseudocode for the DSAGE algorithm.

Original: The model exploitation phase (lines 11–20) is an inner loop that leverages existing QD optimizationalgorithms and the predictions of the deep surrogate model to build an archive – referred to as thesurrogate archive – of solutions.
Revised: The model exploitation phase (lines 11–20) is an inner loop that leverages existing QD optimization algorithms and the predictions of the deep surrogate model to build an archive – referred to as the surrogate archive – of solutions.

Original: The first step of this phase is to query a list of B candidate solutionsthrough the QD algorithm’s ask method.
Revised: The first step of this phase is to query a list of B candidate solutions through the QD algorithm’s ask method.

Original: These solutions are environment parameters, e.g., latentvectors of a GAN, which are passed through the environment generator, e.g., a GAN, to create anenvironment (line 15).
Revised: These solutions are environment parameters, e.g., latent vectors of a GAN, which are passed through the environment generator, e.g., a GAN, to create an environment (line 15).

Original: Next, we make predictions with the surrogate model.
Revised: Next, we make predictions with the surrogate model.

Original: The surrogate modelfirst predicts data representing the agent’s behavior, e.g., the probability of occupying each discretizedtile in the environment (line 16), referred to as “ancillary agent behavior data” ( y ).
Revised: The surrogate model first predicts data representing the agent’s behavior, e.g., the probability of occupying each discretized tile in the environment (line 16), referred to as “ancillary agent behavior data” ( y ).

Original: The predictedancillary agent behavior data ( ˆ y ) then guides the surrogate model’s downstream prediction of theobjective ( ˆ f ) and the measure values ( ˆ m ) (line 17).
Revised: The predicted ancillary agent behavior data ( ˆ y ) then guides the surrogate model’s downstream prediction of the objective ( ˆ f ) and the measure values ( ˆ m ) (line 17).

Original: Finally, the QD algorithm’s tell method adds thesolution to the surrogate archive based on the predicted objective and measure values.
Revised: Finally, the QD algorithm’s tell method adds the solution to the surrogate archive based on the predicted objective and measure values.

Original: Note that since DSAGE is independent of the QD algorithm, the ask and tell methods abstract out the
Revised: Note that since DSAGE is independent of the QD algorithm, the ask and tell methods abstract out the

Original: QD algorithm’s details.
Revised: QD algorithm’s details.

Original: For example, when the QD algorithm is MAP-Elites or CMA-ME, tell addssolutions if the cell in the measure space that they belong to is empty or if the existing solution inthat cell has a lower objective value.
Revised: For example, when the QD algorithm is MAP-Elites or CMA-ME, tell adds solutions if the cell in the measure space that they belong to is empty or if the existing solution in that cell has a lower objective.

Original: For CMA-ME, tell also includes updating internal CMA-ESparameters.
Revised: For CMA-ME, tell also includes updating internal CMA-ES parameters.

Original: The agent simulation phase (lines 21–28) inserts a subset of solutions from the surrogate archive intothe ground-truth archive.
Revised: The agent simulation phase (lines 21–28) inserts a subset of solutions from the surrogate archive into the ground-truth archive.

Original: This phase begins by selecting the subset of solutions from the surrogatearchive (line 21).
Revised: This phase begins by selecting the subset of solutions from the surrogate archive (line 21).

Original: The selected solutions are evaluated by generating the corresponding environment(line 23) and simulating a fixed agent to obtain the true objective and measure values, as well as ancillary agent behavior data (line 24).
Revised: The selected solutions are evaluated by generating the corresponding environment (line 23) and simulating a fixed agent to obtain the true objective and measure values, as well as ancillary agent behavior data (line 24).

Original: Evaluation data is appended to the dataset, and solutions thatimprove their corresponding cell in the ground-truth archive are added to that archive (lines 25, 26).
Revised: Evaluation data is appended to the dataset, and solutions that improve their corresponding cell in the ground-truth archive are added to that archive (lines 25, 26).

Original: In the model improvement phase (line 29), the surrogate model is trained in a self-supervised mannerthrough the supervision provided by the agent simulations and the ancillary agent behavior data.
Revised: In the model improvement phase (line 29), the surrogate model is trained in a self-supervised manner through the supervision provided by the agent simulations and the ancillary agent behavior data.

Original: The algorithm is initialized by generating random solutions and simulating the agent in the corresponding environments (lines 2-8).
Revised: The algorithm is initialized by generating random solutions and simulating the agent in the corresponding environments (lines 2-8).

Original: Subsequently, every outer iteration (lines 10-30) consists of modelexploitation followed by agent simulation and ending with model improvement.
Revised: Subsequently, every outer iteration (lines 10-30) consists of model exploitation followed by agent simulation and ending with model improvement.

Revised: Input: N : Maximum number of evaluations, n rand : Number of initial random solutions,

Original:  do 
Revised: N exploit : Number of iterations in the model exploitation phase, B : Batch size for the model exploitation QD optimizer Output: Final version of the ground-truth archive A gt 1 Initialize the ground-truth archive A gt , the dataset D , and the deep surrogate model sm 2 Θ ← generate_random_solutions ( n rand ) 3 for θ ∈ Θ do env ← g ( θ ) f, m , y ← evaluate ( env ) D ← D ∪ ( θ , f, m , y )

Original:  8 end  do  do  do 
Revised: A gt ← add_solution ( A gt , ( θ , f, m ) ) 8 end 9 evals ← n rand 10 while evals < N do Initialize a QD optimizer qd with the surrogate archive A surrogate for itr ∈ { 1 , 2 , . . , N exploit } do Θ ← qd.ask B ) for θ ∈ Θ do env ← g ( θ )

Original:  end 
Revised: ˆ y ← .predict ˆ f, ˆ m ← sm.predict ( env, ˆ y ) qd.tell ( θ , ˆ f, ˆ m ) end end Θ ← select_solutions ( A surrogate ) for θ ∈ Θ do env ← g ( θ ) f, m , y ← evaluate ( env )

Revised: D ← D ∪ ( θ , f, m , y )

Revised: A gt ← add_solution ( A gt , ( θ , f, m

Revised: ) evals ← evals + end sm.train

Original:  Model Exploitation
Revised: D ) Model Exploitation

Original: Agent Simulation
Revised: Agent Simulation

Original: Model Improvement 30 end
Revised: Model Improvement 30 end

Original: Self-supervised prediction of ancillary agent behavior data.
Revised: Self-supervised prediction of ancillary agent behavior data.

Original: By default, a surrogate modeldirectly predicts the objective and measure values based on the initial state of the environment andthe agent (provided in the form of a one-hot encoded image).
Revised: By default, a surrogate model directly predicts the objective and measure values based on the initial state of the environment and the agent (provided in the form of a one-hot encoded image).

Original: However, we anticipate that directprediction will be challenging in some domains, as it requires understanding the agent’s trajectory inthe environment.
Revised: However, we anticipate that direct prediction will be challenging in some domains, as it requires understanding the agent’s trajectory in the environment.

Original: Thus, we provide additional supervision to the surrogate model in DSAGE via atwo-stage self-supervised process.
Revised: Thus, we provide additional supervision to the surrogate model in DSAGE via a two-stage self-supervised process.

Original: First, a deep neural network predicts ancillary agent behavior data.
Revised: First, a deep neural network predicts ancillary agent behavior data.

Original: In our work, we obtain this databy recording the expected number of times the agent visits each discretized tile in the environment,resulting in an “occupancy grid.”
Revised: In our work, we obtain this data by recording the expected number of times the agent visits each discretized tile in the environment, resulting in an “occupancy grid.”

Original: We then concatenate the predicted ancillary information, i.e., thepredicted occupancy grid, with the one-hot encoded image of the environment and pass them throughanother deep neural network to obtain the predicted objective and measure values.
Revised: We then concatenate the predicted ancillary information, i.e., the predicted occupancy grid, with the one-hot encoded image of the environment and pass them through another deep neural network to obtain the predicted objective and measure values.

Original: We use CNNs for both predictors and includedetails about the architecture  in Appendix B.
Revised: We use CNNs for both predictors and include architecture details in Appendix B.

Original: As a baseline, we compareour model with a CNN that directly predicts the objective and measure values without the help ofancillary data.
Revised: As a baseline, we compare our model with a CNN that directly predicts the objective and measure values without the help of ancillary data.

Original: Downsampling to select solutions from the surrogate archive.
Revised: Downsampling to select solutions from the surrogate archive.

Original: After the model exploitation phase,the surrogate archive is populated with solutions that were predicted to be high-performing anddiverse.
Revised: After the model exploitation phase, the surrogate archive is populated with solutions that were predicted to be high-performing and diverse.

Original: Hence, a basic selection mechanism (line 21) would select all solutions from the surrogatearchive, identical to DSA-ME [19].
Revised: Hence, a basic selection mechanism (line 21) would select all solutions from the surrogate archive, identical to DSA-ME [19].

Original: However, if the surrogate archive is overly populated, fullselection may result in a large number of ground-truth evaluations per outer-loop iteration, leadingto fewer outer loops and less surrogate model training.
Revised: However, if the surrogate archive is overly populated, full selection may result in a large number of ground-truth evaluations per outer-loop iteration, leading to fewer outer loops and less surrogate model training.

Original: To balance the trade-off between evaluatingsolutions from the surrogate archive and training the surrogate model, we only select a subset ofsolutions for evaluation by downsampling the surrogate archive.
Revised: To balance the trade-off between evaluating solutions from the surrogate archive and training the surrogate model, we only select a subset of solutions for evaluation by downsampling the surrogate archive.

Original: Downsampling uniformly dividesthe surrogate archive into sub-regions of cells and selects a random solution from each area.
Revised: Downsampling uniformly divides the surrogate archive into sub-regions of cells and selects a random solution from each area.

Original:  Domains
Revised:  Domains

Original: We test our algorithms in two benchmark domains from previous work: a Maze domain [3, 4] witha trained ACCEL agent [4] and a Mario domain [82, 16] with an A* agent [22].
Revised: We test our algorithms in two benchmark domains from prior work: a Maze domain [82, 3, 4] with a trained ACCEL agent [4] and a Mario domain [83, 16] with an A* agent [22].

Original: We select thesedomains because, despite their relative simplicity (each environment is represented as a 2D grid oftiles), agents in these environments exhibit complex and diverse behaviors.
Revised: We select these domains because, despite their relative simplicity (each environment is represented as a 2D grid of tiles), agents in these environments exhibit complex and diverse behaviors.

Original: Appendix A containsadditional details about the environments.
Revised: In the Maze domain, we directly search for different mazes, with the QD algorithm returning the layout of the maze. In the Mario domain, we search for latent codes that are passed through a pre-trained GAN, similar to the corresponding previous work. We select the objective and measure functions as described below. Since the agent or the environment dynamics are stochastic in each domain, we average the objective and measure values overepisodes in the Maze domain and 5 episodes in the Mario domain.

Original: Maze.
Revised: Maze.

Original: The environment generator directly outputs a 16 × 16 map of walls and empty cells whichdefine a MiniGrid environment [83]. The starting cell and goal cell of the maze are setto the pairof reachable empty cells that are furthest apart in this map, as identified by the Floyd-Warshallalgorithm [84]. The agent needs to reach the goal cell from the starting cell within a specified timelimit. We select the best trained ACCEL [4] agent out of four random seeds.
Revised: We set a binary objective function f

Original: Since we wish to generate valid environments, we set a binary objective function f that is 1 ifthe generated environment is solvable, i.e., if the goal cell is reachable by the starting cell, and 0otherwise.
Revised: that is 1 if the generated environment is solvable and 0 otherwise, indicating the validity of the environment.

Original: Furthermore, we wish to generate environments that vary with respect to the challengethey provide to the agent,thus we selected as measures (1) number of wall cells (ranges from 0 to256), and (2) mean agent path length (ranges from 0 to 648, where 648 indicates a failure to reachthe goal).
Revised: Since we wish to generate visually diverse levels that offer a range of difficulty level for the agent, we select as measures (1) number of wall cells (range: [0 , 256] ), and (2) mean agent path length (range: [0 , 648] , where 648 indicates a failure to reach the goal).

Original: Since the ACCEL agent’s policy is stochastic, we average the path length over 50 episodes(validity of the level and the number of wall cells do not depend on the agent).

Original: Mario.
Revised: Mario.

Original: Drawing upon prior work [16, 58], we generate environments for the Mario AI Framework [85,21] by passing latent vectors to a GAN which was pre-trained on human-authored levels.

Original: Each leveloutput by the GAN’s generator is a 16 × 56 grid of tiles, where each tile can be one of 17 differentobjects.

Original: In each environment, we run the A* agent developed by Robin Baumgarten [22].

Original: Since we wish to generate playable levels, we set the objective as the completion rate , i.e., theproportion of the level that the agent completes before dying.
Revised: Since we wish to generate playable levels, we set the objective as the completion rate , i.e., the proportion of the level that the agent completes before dying.

Original: We additionally want to generateenvironments that result in qualitatively different agent behaviors, thus we selected as measures: (1)sky tiles , the number of tiles of a certain type that are in the top half of the 2D grid (ranges from 0 to150), (2) number of jumps , the number of times that the A* agent jumps during its execution (rangesfrom 0 to 100).
Revised: We additionally want to generate environments that result in qualitatively different agent behaviors, thus we selected as measures: (1) sky tiles , the number of tiles of a certain type that are in the top half of the 2D grid (range: [0 , 150] ), (2) number of jumps , the number of times that the A* agent jumps during its execution (range: [0 , 100] ).

Original: Since the environment dynamics are stochastic, for the ground-truth evaluations thecompletion and number of jumps are averaged over 5 episodes (the number of sky tiles does notdepend on the agent’s execution).
Revised: See Appendix A for further environment details.

Original:  Experiments 
Revised:  Experiments 6.1 Experiment Design

Original: 6.1 Experiment Design

Original: Independent variables.
Revised: Independent variables.

Original: In each domain (Maze and Mario), we follow a between-groups design,where the independent variable is the algorithm.
Revised: In each domain (Maze and Mario), we follow a between-groups design, where the independent variable is the algorithm.

Original: We test the following algorithms:
Revised: We test the following algorithms:

Original: DSAGE :
Revised: DSAGE :

Original: The proposed algorithm that includes predicting ancillary agent behavior data and downsam- pling the surrogate archive (Sec. 4).
Revised: The proposed algorithm that includes predicting ancillary agent behavior data and downsampling the surrogate archive (Sec. 4).

Original: DSAGE-Only Anc : The proposed algorithm with ancillary data prediction and no downsampling, i.e.,
Revised: DSAGE-Only Anc : The proposed algorithm with ancillary data prediction and no downsampling, i

Original: [Figure]
Revised: e., selecting all solutions from the surrogate archive. DSAGE-Only Down : The proposed algorithm with downsampling and no ancillary data prediction.

Original: [Figure]
Revised: [Figure]

Original: version of the proposed algorithm that selects all solutions from thesurrogate archive and does not predict ancillary data.

Original: Baseline QD : The QD algorithm without surrogate assistance.
Revised: Baseline QD : The QD algorithm without surrogate assistance.

Original: We follow previous work [16] anduse CMA-ME for the Mario domain.
Revised: We follow previous work [16] and use CMA-ME for the Mario domain.

Original: Since CMA-ME operates only in continuous spaces, we use
Revised: Since CMA-ME operates only in continuous spaces, we use

Original: MAP-Elites in the discrete Maze domain.
Revised: MAP-Elites in the discrete Maze domain.

Original: Domain Randomization (DR) [86–88] : Algorithm that generates and evaluate random solutions,i.e. wall locations for the maze domain and the latent code to pass through the GAN for the Mariodomain, respectively.
Revised: Domain Randomization (DR) [84–86] : Algorithm that generates and evaluates random solutions, i.e., wall locations in the maze domain and the latent code to pass through the GAN in the Mario domain.

Original: Dependent variables.
Revised: Dependent variables.

Original: We measure the quality and diversity of the solutions with the QD-scoremetric [26](Eq. 1).
Revised: We measure the quality and diversity of the solutions with the QD-score metric [26](Eq. 1).

Original: As an additional metric of diversity, we also report the archive coverage.
Revised: As an additional metric of diversity, we also report the archive coverage.

Original: We runeach algorithm for 5 trials in each domain.
Revised: We run each algorithm for 5 trials in each domain.

Original: Hypothesis.
Revised: Hypothesis.

Original: We hypothesize that DSAGE will result in a better QD-score than DSAGE Basic in alldomains, which in turn will result in better performance than the baseline QD algorithm.
Revised: We hypothesize that DSAGE will result in a better QD-score than DSAGE Basic in all domains, which in turn will result in better performance than the baseline QD algorithm.

Original: DSAGE,
Revised: DSAGE,

Original: DSAGE Basic, and the baseline QD algorithm will all exceed DR.
Revised: DSAGE Basic, and the baseline QD algorithm will all exceed DR.

Original: We base this hypothesis on previouswork [25, 17] which shows that QD algorithms outperform random sampling in a variety of domains,as well as previous work [18, 19] which shows that surrogate-assisted MAP-Elites outperformsstandard MAP-Elites in design optimization and Hearthstone domains.
Revised: We base this hypothesis on previous work [25, 17] which shows that QD algorithms outperform random sampling in a variety of domains, as well as previous work [18, 19] which shows that surrogate-assisted MAP-Elites outperforms standard MAP-Elites in design optimization and Hearthstone domains.

Original: Furthermore, we expect thatthe additional supervision through ancillary agent behavior data and downsampling will result in
Revised: Furthermore, we expect that the additional supervision through ancillary agent behavior data and downsampling will result in

Original: DSAGE performing significantly better than DSAGE Basic.
Revised: DSAGE performing significantly better than DSAGE Basic.

Original: 6.2 Analysis
Revised: 6.2 Analysis

Original: Fig.
Revised: Fig.

Original: 2 summarizes the results obtained by the five algorithms on the Maze and the Mario domains.
Revised: 2 summarizes the results obtained by the six algorithms on the Maze and the Mario domains.

Original: One-way ANOVA tests showed a significant effect of the algorithm on the QD-score for the Maze( F (5 , 24) = 430 .
Revised: One-way ANOVA tests showed a significant effect of the algorithm on the QD-score for the Maze ( F (5 , 24) = 430 .

Original: 98 , p < 0 .
Revised: 98 , p < 0 .

Original: and Mario ( F (5 , 24) = 238 .
Revised: and Mario ( F (5 , 24) = 238 .

Original: 09 , p < 0 .
Revised: 09 , p < 0 .

Original: 001 ) domains.
Revised: 001 ) domains.

Original: Post-hoc pairwise comparisons with Bonferroni corrections showed that DSAGE outperformed
Revised: Post-hoc pairwise comparisons with Bonferroni corrections showed that DSAGE outperformed

Original: DSAGE Basic, Baseline QD, and DR in both the Maze and the Mario domains ( p < 0 . 001 ).
Revised: DSAGE Basic, Baseline QD, and DR in both the Maze and the Mario domains ( p < 0 . 001 ).

Original: Additionally, DSAGE Basic outperformed MAP-Elites and DR in the Maze domain ( p < 0 .
Revised: Additionally, DSAGE Basic outperformed MAP-Elites and DR in the Maze domain ( p < 0 .

Original: 001) ,while it performed significantly worse than the QD counterpart, CMA-ME, in the Mario domain ( p =0 . 003) .
Revised: 001) , while it performed significantly worse than the QD counterpart, CMA-ME, in the Mario domain ( p = 0 . 003) .

Original: Finally, Baseline QD outperformed DR in both the Maze and Mario domains ( p < 0 . 001 ).
Revised: Finally, Baseline QD outperformed DR in both the Maze and Mario domains ( p < 0 . 001 ).

Original: These results show that deep surrogate assisted generation of environments results in significantimprovements compared to quality diversity algorithms without surrogate assistance.
Revised: These results show that deep surrogate assisted generation of environments results in significant improvements compared to quality diversity algorithms without surrogate assistance.

Original: They also show[Table] [Table] that adding ancillary agent behavior data and downsampling is important in both domains.
Revised: They also show that adding ancillary agent behavior data and downsampling is important in both domains.

Original: Withoutthese components, DSAGE Basic has limited or no improvement compared to the QD algorithmwithout surrogate assistance.
Revised: Without these components, DSAGE Basic has limited or no improvement compared to the QD algorithm [Table] [Table] without surrogate assistance.

Original: Additionally, domain randomization is significantly worse than DSAGEas well as the baselines.
Revised: Additionally, domain randomization is significantly worse than DSAGE as well as the baselines.

Original: The archive coverage and consequently the QD score is negligible in the
Revised: The archive coverage and consequently the QD-score is negligible in the

Original: Mario domain since randomly sampled latent codes led to little diversity in the levels
Revised: Mario domain since randomly sampled latent codes led to little diversity in the levels

Original: [Table]
Revised: [Table]

Original: Table 1 shows another metric of the speed-up provided by DSAGE: the number of evaluations (agentsimulations) required to reach a fixed QD score.
Revised: Table 1 shows another metric of the speed-up provided by DSAGE: the number of evaluations (agent simulations) required to reach a fixed QD-score.

Original: We set this fixed QD score to be 10480.8 in the Mazedomain and 1306.11 in the Mario domain, which are the mean QD scores of MAP-Elites andDSAGE
Revised: We set this fixed QD-score to be 10480.8 in the Maze domain and 1306.11 in the Mario domain, which are the mean QD-scores of MAP-Elites and

Original:  Basic, respectively, in those domains.
Revised: DSAGE Basic, respectively, in those domains.

Original: DSAGE reaches these QD scores faster than the baselines do.
Revised: DSAGE reaches these QD-scores faster than the baselines do.

Original: To assess the quality of the trained surrogate model, we create a combined dataset consisting of datafrom one run of each surrogate assisted algorithm.
Revised: To assess the quality of the trained surrogate model, we create a combined dataset consisting of data from one run of each surrogate assisted algorithm.

Original: We use this dataset to evaluate the surrogatemodels trained from separate runs of DSAGE and its variants.
Revised: We use this dataset to evaluate the surrogate models trained from separate runs of DSAGE and its variants.

Original: Table 2 shows the mean absolute error(MAE) of the predictions by the surrogate models.
Revised: Table 2 shows the mean absolute error (MAE) of the predictions by the surrogate models.

Original: The model learned by DSAGE Basic fails topredict the agent-based measures well.
Revised: The model learned by DSAGE Basic fails to predict the agent-based measures well.

Original: It has an MAE of 157.69 for the mean agent path length in
Revised: It has an MAE of 157.69 for the mean agent path length in

Original: Maze and MAE = 10.71 for the number of jumps in Mario.
Revised: Maze and MAE = 10.71 for the number of jumps in Mario.

Original: In contrast, the model learned by DSAGEmakes more accurate predictions, with MAE = 96.58 forthe mean agent path length and MAE =
Revised: In contrast, the model learned by DSAGE makes more accurate predictions, with MAE = 96.58 for mean agent path length and MAE =

Original: 7.16 forthe number of jumps.
Revised: 7.16 for number of jumps.

Original: We providethe detailed results of the surrogate model predictions in Appendix B.1.
Revised: We provide detailed results of the surrogate model predictions in Appendix B.1.

Original: 6.3 Ablation Study
Revised: 6.3 Ablation Study

Original: Sec. 4 describes two key components of DSAGE: (1) self-supervised prediction of ancillary agentbehavior data, and (2) downsampling to select solutions from the surrogate archive.
Revised: Sec. 4 describes two key components of DSAGE: (1) self-supervised prediction of ancillary agent behavior data, and (2) downsampling to select solutions from the surrogate archive.

Original: We perform anablation study by treating the inclusion of ancillary data prediction (ancillary data / no ancillary data)and the method of selecting solutions from the surrogate archive (downsampling / full selection) asindependent variables.
Revised: We perform an ablation study by treating the inclusion of ancillary data prediction (ancillary data / no ancillary data) and the method of selecting solutions from the surrogate archive (downsampling / full selection) as independent variables.

Original: A two-way ANOVA for each domain showed no significant interaction effects.
Revised: A two-way ANOVA for each domain showed no significant interaction effects.

Original: We perform a main effects analysis for each independent variable.
Revised: We perform a main effects analysis for each independent variable.

Original: Inclusion of ancillary data prediction.
Revised: Inclusion of ancillary data prediction.

Original: A main effects analysis for the inclusion of ancillary dataprediction showed that algorithms that predict ancillary agent behavior data (DSAGE, DSAGEOnly Anc) performed significantly better than their counterparts with no ancillary data prediction(DSAGE-Only Down, DSAGE Basic) in both domains ( p < 0 .001 ).
Revised: A main effects analysis for the inclusion of ancillary data prediction showed that algorithms that predict ancillary agent behavior data (DSAGE, DSAGE-Only Anc) performed significantly better than their counterparts with no ancillary data prediction (DSAGE-Only Down, DSAGE Basic) in both domains ( p < 0 .

Original: Fig.
Revised: Fig.

Original: 2 shows that predicting ancillary agent behavior data also resulted in a larger mean coverage for
Revised: 2 shows that predicting ancillary agent behavior data also resulted in a larger mean coverage for

Original: Maze, while it has little or no improvement for Mario.
Revised: Maze, while it has little or no improvement for Mario.

Original: Additionally, as shown in Table 2, predictingancillary agent behavior data helped improve the prediction of the mean agent path length in the
Revised: Additionally, as shown in Table 2, predicting ancillary agent behavior data helped improve the prediction of the mean agent path length in the

Original: Maze domain but provided little improvement to the prediction of the number of jumps in the Mariodomain.
Revised: Maze domain but provided little improvement to the prediction of the number of jumps in the Mario domain.

Original: The reason is that in the Maze domain, the mean agent path length is a scaled version of the sum of the agent’s tile occupancy frequency, hence the two-stage process which predicts theoccupancy grid first is essential for improving the accuracy of the model.
Revised: The reason is that in the Maze domain, the mean agent path length is a scaled version of the sum of the agent’s tile occupancy frequency, hence the two-stage process which predicts the occupancy grid first is essential for improving the accuracy of the model.

Original: On the other hand, thepresence of a jump in Mario depends not only on cell occupancy, but also on the structure of the leveland the sequence of the occupied cells.
Revised: On the other hand, the presence of a jump in Mario depends not only on cell occupancy, but also on the structure of the level and the sequence of the occupied cells.

Original: Method of selecting solutions from the surrogate archive.
Revised: Method of selecting solutions from the surrogate archive.

Original: A main effects analysis for the methodof selecting solutions from the surrogate archive showed that the algorithms with downsampling(DSAGE, DSAGE-Only Down) performed significantly better than their counterparts with no downsampling (DSAGE-Only Anc, DSAGE-Basic) in both domains ( p < 0 . 001 ).
Revised: A main effects analysis for the method of selecting solutions from the surrogate archive showed that the algorithms with downsampling (DSAGE, DSAGE-Only Down) performed significantly better than their counterparts with no downsampling (DSAGE-Only Anc, DSAGE Basic) in both domains ( p < 0 . 001 ).

Original: A major advantage of downsampling is that it decreases the number of ground-truth evaluations ineach outer iteration.
Revised: A major advantage of downsampling is that it decreases the number of ground-truth evaluations in each outer iteration.

Original: Thus, for a fixed evaluation budget, downsampling results in a greater numberof outer iterations.
Revised: Thus, for a fixed evaluation budget, downsampling results in a greater number of outer iterations.

Original: For instance, in the Maze domain, runs without downsampling had only 6-7outer iterations, while runs with downsampling had approximately 220 outer iterations.
Revised: For instance, in the Maze domain, runs without downsampling had only 6outer iterations, while runs with downsampling had approximately 220 outer iterations.

Original: More outeriterations leads to more training and thus higher accuracy of the surrogate model.
Revised: More outer iterations leads to more training and thus higher accuracy of the surrogate model.

Original: In turn, a moreaccurate surrogate model will generate a better surrogate archive in the inner loop.
Revised: In turn, a more accurate surrogate model will generate a better surrogate archive in the inner loop.

Revised: We include an ablation in Appendix D to test between two possible explanations for why having more outer iterations helps with performance: (1) larger number of training epochs, (2) more updates to the dataset allowing the surrogate model to iteratively correct its own errors.

Revised: We observed that iterative correction accounted for most of the performance increase with downsampling.

Original: The second advantage of downsampling is that it selects solutions evenly from all regions of themeasure space, thus creating a more balanced dataset.
Revised: The second advantage of downsampling is that it selects solutions evenly from all regions of the measure space, thus creating a more balanced dataset.

Original: This helps train the surrogate model in parts ofthe measure space that are not frequently visited.
Revised: This helps train the surrogate model in parts of the measure space that are not frequently visited.

Original: We include an additional baseline in Appendix Din which we select a subset of solutions uniformly at random from the surrogate archive instead ofdownsampling.
Revised: We include an additional baseline in Appendix E in which we select a subset of solutions uniformly at random from the surrogate archive instead of downsampling.

Original: We observe that downsampling has a slight advantage over uniform random samplingin the Maze domain.
Revised: We observe that downsampling has a slight advantage over uniform random sampling in the Maze domain.

Original: Furthermore, if instead of downsampling we sampled multiple solutions from nearby regions of thesurrogate archive, the prediction errors could cause the solutions to collapse to a single cell in theground-truth archive, resulting in many solutions being discarded.
Revised: Furthermore, if instead of downsampling we sampled multiple solutions from nearby regions of the surrogate archive, the prediction errors could cause the solutions to collapse to a single cell in the ground-truth archive, resulting in many solutions being discarded.

Original: Overall, our ablation study shows that both predicting the occupancy grid as ancillary data anddownsampling the surrogate archive independently help improve the performance of DSAGE.
Revised: Overall, our ablation study shows that both predicting the occupancy grid as ancillary data and downsampling the surrogate archive independently help improve the performance of DSAGE.

Original: 6.4 Qualitative Results
Revised: 6.4 Qualitative Results

Original: Fig.
Revised: Fig.

Original: 3 and Fig.
Revised: 3 and Fig.

Original: 4 show example environments generated by DSAGE in the Maze and the Mariodomains.
Revised: 4 show example environments generated by DSAGE in the Maze and Mario domains.

Original: Having the mean agent path length as a measure in the Maze domain results in environments ofvarying difficulty for the ACCEL agent.
Revised: Having the mean agent path length as a measure in the Maze domain results in environments of varying difficulty for the ACCEL agent.

Original: For instance, we observe that the environment in Fig. 3(a)has very few walls, yet the ACCEL agent gets stuck in the top half of the maze and is unable to findthe goal within the allotted time.
Revised: For instance, we observe that the environment in Fig. 3(a) has very few walls, yet the ACCEL agent gets stuck in the top half of the maze and is unable to find the goal within the allotted time.

Original: On the other hand, the environment in Fig. 3(d) is cluttered andthere are multiple dead-ends, yet the ACCEL agent is able to reach the goal.
Revised: On the other hand, the environment in Fig. 3(d) is cluttered and there are multiple dead-ends, yet the ACCEL agent is able to reach the goal.

Original: Fig.
Revised: Fig.

Original: 4 shows that the generated environments result in qualitatively diverse behaviors for the Marioagent too.
Revised: 4 shows that the generated environments result in qualitatively diverse behaviors for the Mario agent too.

Original: Level (b) only has a few sky tiles and is mostly flat, resulting in a small number of jumps.
Revised: Level (b) only has a few sky tiles and is mostly flat, resulting in a small number of jumps.

Original: Level (c) has a “staircase trap” on the right side, forcing the agent to perform continuous jumps toescape and complete the level.
Revised: Level (c) has a “staircase trap” on the right side, forcing the agent to perform continuous jumps to escape and complete the level.

Original: We include videos of the playthroughs in the supplemental material.
Revised: We include videos of the playthroughs in the supplemental material.

Original:  Societal Impacts
Revised:  Societal Impacts

Original: By introducing surrogate models into quality diversity algorithms, we can efficiently generateenvironments that result in diverse agent behaviors.
Revised: By introducing surrogate models into quality diversity algorithms, we can efficiently generate environments that result in diverse agent behaviors.

Original: While we focused on an RL agent in a Mazedomain and a planning agent in a Mario game domain, our method can be applied to a variety ofagents and domains.
Revised: While we focused on an RL agent in a Maze domain and a symbolic agent in a Mario game domain, our method can be applied to a variety of agents and domains.

Original: This can help with testing the robustness of agents, attaining insights about theirbehavior, and discovering edge cases before real-world deployment.
Revised: This can help with testing the robustness of agents, attaining insights about their behavior, and discovering edge cases before real-world deployment.

Original: Furthermore, we anticipate thatin the future, closing the loop between environment generation and agent training can improve theability of agents to generalize to new settings and thus increase their widespread use.
Revised: Furthermore, we anticipate that in the future, closing the loop between environment generation and agent training can improve the ability of agents to generalize to new settings and thus increase their widespread use.

Original: Our work may also have negative impacts.
Revised: Our work may also have negative impacts.

Original: Training agents in diverse environments can be consideredas a step towards open-ended evolution [89], which raises concerns about the predictability and safetyof the emergent agent behaviors [90, 91].
Revised: Training agents in diverse environments can be considered as a step towards open-ended evolution [87], which raises concerns about the predictability and safety of the emergent agent behaviors [88, 89].

Original: Discovering corner cases that result in unwanted behaviorsor catastrophic failures may also be used maliciously to reveal vulnerabilities in deployed agents [92].
Revised: Discovering corner cases that result in unwanted behaviors or catastrophic failures may also be used maliciously to reveal vulnerabilities in deployed agents [90].

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original:  Limitations and Future Work
Revised:  Limitations and Future Work

Original: Automatic environment generation is a rapidly growing research area with a wide range of applications,including designing video game levels [46, 53, 54], training and testing autonomous agents [74, 76,1, 3, 4]
Revised: Automatic environment generation is a rapidly growing research area with a wide range of applications, including designing video game levels [46, 53, 54], training and testing autonomous agents [74, 76, 1, 3, 4]

Original: [Figure]
Revised: [Figure]

Original: and discovering failures in human-robot interaction [17, 16].
Revised: and discovering failures in human-robot interaction [17, 16].

Original: We introduce the DSAGEalgorithm, which efficiently generates a diverse collection of environments via deep surrogate modelsof agent behavior
Revised: We introduce the DSAGE algorithm, which efficiently generates a diverse collection of environments via deep surrogate models of agent behavior

Original: [Figure]
Revised: [Figure]

Original: Our paper has several limitations.
Revised: Our paper has several limitations.

Original: First, occupancy grid prediction does not encode temporalinformation about the agent.
Revised: First, occupancy grid prediction does not encode temporal information about the agent.

Original: On one hand, this prediction allows us to avoid the compounding errorproblem of model-based RL [93]. On the other hand, forgoing temporal information makes it harderto predict some behaviors, such as the number of jumps in Mario.
Revised: While this prediction allows us to avoid the compounding error problem of model-based RL [91], forgoing temporal information makes it harder to predict some behaviors, such as the number of jumps in Mario.

Original: We will explore this trade-off infuture work.
Revised: We will explore this trade-off in future work.

Original: Furthermore, we have studied 2D domains where a single ground-truth evaluation lasts between afew seconds and a few minutes.
Revised: Furthermore, we have studied 2D domains where a single ground-truth evaluation lasts between a few seconds and a few minutes.

Original: We are excited about the use of surrogate models to predict theperformance of agents in more complex domains with expensive, high-fidelity simulations [94]
Revised: We are excited about the use of surrogate models to predict the performance of agents in more complex domains with expensive, high-fidelity simulations [92].

Original: Checklist
Revised: Acknowledgments and Disclosure of Funding

Revised: This work was partially funded by NSF CAREER (#2145077) and NSF GRFP (#DGE-1842487). One of the GPUs used in the experiments was awarded by the NVIDIA Academic Hardware Grant. We thank J. Parker-Holder et al.

Revised: , the authors of the ACCEL agent, for providing a pre-trained model of the agent for our experiments.

Revised: We also thank Ya-Chuan Hsu for providing invaluable feedback.

Original: For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’scontributions and scope?
Revised: For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?

Original: [Yes] We claim that our algorithm significantly outperforms QD environment generation algorithms.
Revised: [Yes] We claim that our algorithm significantly outperforms QD environment generation algorithms.

Original: The claim is supported by results and thestatistical tests shown in Sec. 6.2.
Revised: The claim is supported by results and the statistical tests shown in Sec. 6.2.

Original: (b) Did you describe the limitations of your work?
Revised: (b) Did you describe the limitations of your work?

Original: [Yes] We list the limitations of our work in Sec. 8.
Revised: [Yes] We list the limitations of our work in Sec. 8.

Original: (c) Did you discuss any potential negative societal impacts of your work?
Revised: (c) Did you discuss any potential negative societal impacts of your work?

Original: [Yes] We discuss the potential positive and negative impacts of this work in Sec. 
Revised: [Yes] We discuss the potential positive and negative impacts of this work in Sec. 7.

Original: (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
Revised: (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?

Original: [Yes] 
Revised: [Yes] 2.

Original: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
Revised: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?

Original: [N/A] (b) Did you include complete proofs of all theoretical results?
Revised: [N/A] (b) Did you include complete proofs of all theoretical results?

Original: [N/A] 
Revised: [N/A] 3.

Original: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
Revised: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?

Original: [Yes] We provide thecode in the supplemental material for running all the experiments specified in Sec. 6.
Revised: [Yes] We provide the code in the supplemental material for running all the experiments specified in Sec. 6.

Original: The code contains a README file with the instructions to reproduce the experiments.
Revised: The code contains a README file with the instructions to reproduce the experiments.

Original: (b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?
Revised: (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?

Original: [Yes] The main details are presented in Sec. 5 and Sec. 6.
Revised: [Yes] The main details are presented in Sec. 5 and Sec. 6.

Original: Other details are provided in Appendix C and also available in the attached code.
Revised: Other details are provided in Appendix C and also available in the attached code.

Original: (c) Did you report error bars (e.g., with respect to the random seed after running exper- iments multiple times)?
Revised: (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?

Original: [Yes] All results shown in Sec.
Revised: [Yes] All results shown in Sec.

Original: 6.2 and Sec.
Revised: 6.2 and Sec.

Original: 6.3 have the corresponding standard error and the inferences are supported by relevant statisticaltests.
Revised: 6.3 have the corresponding standard error and the inferences are supported by relevant statistical tests.

Original: (d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
Revised: (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?

Original: [Yes] We include the description of the computational resources used for the runs in Appendix C. 
Revised: [Yes] We include the description of the computational resources used for the runs in Appendix C. 4.

Original: If you are using existing assets (e g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
Revised: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?

Original: [Yes] We have used existingopen source code for the Mario simulator and the A* agent, the Maze environment,and QD algorithm implementation.
Revised: [Yes] We have used existing open source code for the Mario simulator and the A* agent, the Maze environment, and QD algorithm implementation.

Original: We have cited the creators either in Sec.
Revised: We have cited the creators either in Sec.

Original: 5 or inAppendix A. We obtained ACCEL agents directly from the original authors with their consent.
Revised: 5 or in Appendix A. We obtained ACCEL agents directly from the original authors with their consent.

Original: We have cited the paper which introduced the algorithm and will acknowledgethe authors for providing the pre-trained agents.
Revised: We have cited the paper which introduced the algorithm and have acknowledged the authors for providing the pre-trained agents.

Original: (b) Did you mention the license of the assets?
Revised: (b) Did you mention the license of the assets?

Original: [Yes] The licenses to the external code andassets are added to the README file in the code that we provide with the supplemental material.
Revised: [Yes] The licenses to the external code and assets are added to the README file in the code that we provide with the supplemental material.

Original: (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [Yes][Table] We include videos of the agent’s playthrough in the supplemental material.
Revised: [Yes] We include videos of the agent’s playthrough in the supplemental material.

Original: (d) Did you discuss whether and how consent was obtained from people whose data you’re[Table] using/curating?
Revised: (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?

Original: [N/A] (e) Did you discuss whether the data you are using/curating contains personally identifiableinformation or offensive content?
Revised: [N/A] (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?

Original: [N/A] 
Revised: [N/A] 5.

Original: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?[N/A]
Revised: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?

Original: [Table]

Original:  (b) Did you describe any potential participant risks, with links to Institutional Review[Table] Board (IRB) approvals, if applicable?[N/A]
Revised: [N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?

Original: [Table] (c) Did you include the estimated hourly wage paid to participants and the total amount[Table] spent on participant compensation?
Revised: [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?

Original: [N/A]
Revised: [N/A]

Original: [Table]