Original: Scaling Covariance Matrix Adaptation MAP-Annealing to High-Dimensional Controllers
Revised: Scaling Covariance Matrix Adaptation MAP-Annealing to High-Dimensional Controllers

Original: Anonymous Author(s)

Original: AfﬁliationAddress email

Original: Abstract
Revised: Abstract

Original: Pre-training a diverse set of robot controllers in simulation has enabled robots toadapt online to damage in robot locomotion tasks.
Revised: Pre-training a diverse set of robot controllers in simulation has enabled robots to adapt online to damage in robot locomotion tasks.

Original: However, ﬁnding diverse, highperforming controllers requires specialized hardware and extensive tuning of a largenumber of hyperparameters.
Revised: However, ﬁnding diverse, high-performing controllers requires specialized hardware and extensive tuning of a large number of hyperparameters.

Original: On the other hand, the Covariance MatrixAdaptation
Revised: On the other hand, the Covariance Matrix

Original:  MAP-Annealing algorithm, an evolution strategies (ES)-based quality diversityalgorithm, does not have these limitations and has been shown to achieve state-ofthe-art performance in standard benchmark domains.
Revised: Adaptation MAP-Annealing algorithm, an evolution strategies (ES)-based quality diversity algorithm, does not have these limitations and has been shown to achieve state-of-the-art performance in standard benchmark domains.

Original: However, CMA-MAE cannotscale to modern neural network controllers due to its quadratic complexity.
Revised: However, CMAMAE cannot scale to modern neural network controllers due to its quadratic complexity.

Original: Weleverage efﬁcient approximation methods in ES to propose three new CMA-MAEvariants that scale to very high dimensions.
Revised: We leverage efﬁcient approximation methods in ES to propose three new CMA-MAE variants that scale to very high dimensions.

Original: Our experiments show that the variantsoutperform ES-based baselines in benchmark robotic locomotion tasks, whilebeing comparable with state-of-the-art deep reinforcement learning-based qualitydiversity algorithms.
Revised: Our experiments show that the variants outperform ES-based baselines in benchmark robotic locomotion tasks, while being comparable with state-of-the-art deep reinforcement learningbased quality diversity algorithms.

Original: Source code and videos are available in the supplementarymaterial.
Revised: Source code and videos are available at https: //scalingcmamae.github.io

Original:  Introduction
Revised:  Introduction

Original: By generating a diverse collection of controllers, we can endow a robot with a wide variety of usefulbehaviors, such as moving an arm into different conﬁgurations [1, 2], walking in every direction [3, 4],pushing an object through many trajectories [5], or traveling to any position in a maze [6, 7, 8].
Revised: By generating a diverse collection of controllers, we can endow a robot with a wide variety of useful behaviors, such as moving an arm into different conﬁgurations [1, 2], walking in every direction [3, 4], pushing an object through many trajectories [5] or traveling to any position in a maze [6, 7, 8].

Original: Having a collection of controllers has proven particularly useful in robotic locomotion, where thecollection enables a walking robot to adapt to damage [9, 10, 11].
Revised: Having a collection of controllers has proven particularly useful in robotic locomotion, where the collection enables a walking robot to adapt to damage [9, 10, 11].

Original: The controllers differ by how longeach foot contacts the ground, such that if one foot is damaged, the robot can select a controller thatdoes not rely on that foot.
Revised: The controllers differ by how long each foot contacts the ground, such that if one foot is damaged, the robot can select a controller that does not rely on that foot.

Original: Searching for these diverse controllers may be viewed as a quality diversity (QD) optimizationproblem [6].
Revised: Searching for these diverse controllers may be viewed as a quality diversity (QD) optimization problem [6].

Original: In QD, the goal is to ﬁnd solutions φ that are diverse with respect to one or moremeasure functions m ( φ ) while maximizing an objective function f ( φ ) .
Revised: In QD, the goal is to ﬁnd solutions φ that are diverse with respect to one or more measure functions m ( φ ) while maximizing an objective function f ( φ ) .

Original: In robotic locomotion, wesearch for controller policies π φ parameterized by φ .
Revised: In robotic locomotion, we search for controller policies π φ parameterized by φ .

Original: Each controller should satisfy a unique outputof the measure function by using its feet in a different manner from the other controllers, whileoptimizing the objective by walking forward quickly.
Revised: Each controller should satisfy a unique output of the measure function by using its feet in a different manner from the other controllers, while optimizing the objective by walking forward quickly.

Original: To solve a QD problem, a QD algorithm must balance two aspects given a limited compute budget:exploring the measure space and optimizing the objective.
Revised: To solve a QD problem, a QD algorithm must balance two aspects given a limited compute budget: exploring the measure space and optimizing the objective.

Original: Furthermore, one of these aspects maybe more important for a given problem.
Revised: Furthermore, one of these aspects may be more important for a given problem.

Original: For example, in locomotion, exploring corresponds toﬁnding new controllers which use the robot’s feet a different amount, while optimizing correspondsto making existing controllers walk faster.
Revised: For example, in locomotion, exploring corresponds to ﬁnding new controllers which use the robot’s feet a different amount, while optimizing corresponds to making existing controllers walk faster.

Original: Prior work [12] shows that optimization is more difﬁcultthan exploration in this domain, as it is trivial to explore the measure space by standing in place andlifting the feet up and down to achieve a new style of foot usage.
Revised: Prior work [12] shows that optimization is more difﬁcult than exploration in this domain, as it is trivial to explore the measure space by standing in place and lifting the feet up and down to achieve a new style of foot usage.

Original: Compute archive improvement
Revised: Compute archive improvement

Original: Adapt and with ES
Revised: Adapt and with ES

Original: [Figure]
Revised: [Figure]

Original: Prior algorithms [13, 12] seem to strike a balance between these two aspects of QD in locomotiontasks, leading to state-of-the-art results.
Revised: Prior algorithms [13, 12] seem to strike a balance between these two aspects of QD in locomotion tasks, leading to state-of-the-art results.

Original: However, these algorithms have practical limitations due totheir dependence on deep reinforcement learning (RL) methods.
Revised: However, these algorithms have practical limitations due to their dependence on deep reinforcement learning (RL) methods.

Original: Namely, they require a GPU, mustperform time-consuming training of a neural network, and have a large number of hyperparameters.
Revised: Namely, they require a GPU, must perform time-consuming training of a neural network, and have a large number of hyperparameters.

Original: Recent work [14] suggests evolution strategies (ES) as a compelling alternative to deep RL methodswhen optimizing a single controller.
Revised: Recent work [14] suggests evolution strategies (ES) as a compelling alternative to deep RL methods when optimizing a single controller.

Original: Compared with deep RL, ESs can run entirely on CPU anddo not require network training, and ESs such as CMA-ES [15] are designed to have almost nohyperparameters.
Revised: Compared with deep RL, ESs can run entirely on CPU and do not require network training, and ESs such as CMA-ES [15] are designed to have almost no hyperparameters.

Original: Given these beneﬁts, prior work [12, 11] has developed QD algorithms based on
Revised: Given these beneﬁts, prior work [12, 11] has developed QD algorithms based on

Original: ESs, but these methods have not yet been able to match the performance of deep RL-based QDmethods.
Revised: ESs, but these methods have not yet been able to match the performance of deep RL-based QD methods.

Original: On the other hand, the recently proposed ES-based Covariance Matrix Adaptation MAP-Annealing(CMA-MAE) algorithm [16] has proven adept at trading off between the exploration and objectiveoptimization aspects of QD.
Revised: On the other hand, the recently proposed ES-based Covariance Matrix Adaptation MAP-Annealing (CMA-MAE) algorithm [16] has proven adept at trading off between the exploration and objective optimization aspects of QD.

Original: Through a single hyperparameter α ∈ [0 , 1] , CMA-MAE can be tuned tooptimize only the objective (as in CMA-ES), or prioritize moving away from previously discoveredhigh-quality solutions (as in CMA-ME [17]), and anywhere in between.
Revised: Through a single hyperparameter α ∈ [0 , 1] , CMA-MAE can be tuned to optimize only the objective (as in CMA-ES), or prioritize moving away from previously discovered high-quality solutions (as in CMA-ME [17]), and anywhere in between.

Original: Selecting any value ofα strictly between 0 and 1 yields theoretical beneﬁts that result in state-of-the-art performance onseveral QD benchmark domains.
Revised: Selecting any value of α strictly between 0 and 1 yields theoretical beneﬁts that result in state-of-the-art performance on several QD benchmark domains.

Original: We hypothesize that because CMA-MAE balances this tradeoff, thealgorithm would excel in robotic locomotion tasks with hard-to-optimize objectives.
Revised: We hypothesize that because CMA-MAE balances this tradeoff, the algorithm would excel in robotic locomotion tasks with hard-to-optimize objectives.

Original: While CMA-MAE excels at moderate dimensional domains, the algorithm is computationally intractable for robotic locomotion due to the high dimensionality of modern neural network controllers.
Revised: While CMA-MAE excels at moderate dimensional domains, the algorithm is computationally intractable for robotic locomotion due to the high dimensionality of modern neural network controllers.

Original: CMA-MAE relies on CMA-ES [15], whose Θ( n 2 ) time complexity per sampled solution, where nis the number of parameters, makes it impossible for CMA-MAE to optimize the thousands or evenmillions of parameters found in such controllers.
Revised: CMA-MAE relies on CMA-ES [15], whose Θ( n 2 ) time complexity per sampled solution, where n is the number of parameters, makes it impossible for CMA-MAE to optimize the thousands or even millions of parameters found in such controllers.

Original: CMA-ES’s complexity arises from how it models the search distribution by maintaining a Gaussianwith a full rank n × n covariance matrix.
Revised: CMA-ES’s complexity arises from how it models the search distribution by maintaining a Gaussian with a full rank n × n covariance matrix.

Original: However, a full rank covariance matrix is often notnecessary, so prior work [18, 19, 20, 21, 22, 23] scales CMA-ES to higher dimensions by replacingthe covariance matrix with sparse approximations.
Revised: However, a full rank covariance matrix is often not necessary, so prior work [18, 19, 20, 21, 22, 23] scales CMA-ES to higher dimensions by replacing the covariance matrix with sparse approximations.

Original: These CMA-ES variants scale to solve a varietyof high-dimensional optimization problems.
Revised: These CMA-ES variants scale to solve a variety of high-dimensional optimization problems.

Original: Thus, our key insight is that we can scale CMA-MAE to high-dimensional controllers by adopting thesame approximations in its CMA-ES components .
Revised: Thus, our key insight is that we can scale CMA-MAE to high-dimensional controllers by adopting the same approximations in its CMA-ES components .

Original: Though our CMA-MAE variants leverage sparsecovariance matrix approximations, we hypothesize these variants will be competitive.
Revised: Though our CMA-MAE variants leverage sparse covariance matrix approximations, we hypothesize these variants will be competitive.

Original: We scale CMA-MAE by proposing three CMA-MAE variants, which we evaluate on robotic locomotion tasks from QDGym [24].
Revised: We scale CMA-MAE by proposing three CMA-MAE variants, which we evaluate on robotic locomotion tasks from QDGym [24].

Original: We show that our variants are the highest-performing QD methodsbased solely on ES.
Revised: We show that our variants are the highest-performing QD methods based solely on ES.

Original: Furthermore, they inherit the aforementioned practical beneﬁts of ES, and theyare comparable to the state-of-the-art deep RL-based QD method PGA-ME [13] on three of fourtasks.
Revised: Furthermore, they inherit the aforementioned practical beneﬁts of ES, and they are comparable to the state-of-the-art deep RL-based QD method PGA-ME [13] on three of four tasks.

Original: We are excited about future applications in other domains, such as robotic manipulation [25]and scenario generation [26].
Revised: We are excited about future applications in other domains, such as robotic manipulation [25] and scenario generation [26].

Original:  Problem Statement
Revised:  Problem Statement

Original: Quality diversity (QD).
Revised: Quality diversity (QD).

Original: Drawing from recent work [27], we deﬁne QD as considering an objectivefunction f ( φ ) and k -dimensional measure function m ( φ ) , 1 where φ ∈ R n is an n -dimensionalsolution.
Revised: Drawing from recent work [27], we deﬁne QD as considering an objective function f ( φ ) and k -dimensional measure function m ( φ ) , 1 where φ ∈ R n is an n -dimensional solution.

Original: The outputs of m form a k -dimensional measure space X .
Revised: The outputs of m form a k -dimensional measure space X .

Original: The QD objective is to ﬁndsolutions φ such that ∀ x ∈ X , m ( φ ) = x and f ( φ ) is maximized.
Revised: The QD objective is to ﬁnd solutions φ such that ∀ x ∈ X , m ( φ ) = x and f ( φ ) is maximized.

Original: Solving this QD objective wouldrequire inﬁnite memory because X is a continuous space, so algorithms based on MAP-Elites [9, 10]relax the QD objective by discretizing X into a tesselation Y of M cells.
Revised: Solving this QD objective would require inﬁnite memory because X is a continuous space, so algorithms based on MAP-Elites [9, 10] relax the QD objective by discretizing X into a tesselation Y of M cells.

Original: Then, the QD objective isto ﬁnd an archive (i.e., a collection)
Revised: Then, the QD objective is to ﬁnd an archive (i.e., a collection)

Original: A containing M elites φ i , where each φ i has measures m ( φ i )corresponding to a cell in Y and f ( φ i ) is maximized.
Revised: A containing M elites φ i , where each φ i has measures m ( φ i ) corresponding to a cell in Y and f ( φ i ) is maximized.

Original: We express this relaxed QD objective asmax φ 1 ..M (cid:80) Mi
Revised: We express this relaxed QD objective as max φ 1 ..M (cid:80) Mi

Original: =1 f ( φ i ) .
Revised: =1 f ( φ i ) .

Original: Quality diversity for reinforcement learning (QD-RL).
Revised: Quality diversity for reinforcement learning (QD-RL).

Original: As deﬁned in prior work [12], QD-RL isa special instance of QD in which φ parameterizes a reinforcement learning (RL) agent’s policy π φand the objective is the expected discounted return of the agent.
Revised: As deﬁned in prior work [12], QD-RL is a special instance of QD in which φ parameterizes a reinforcement learning (RL) agent’s policy π φ and the objective is the expected discounted return of the agent.

Original: QD-RL extends Markov Decision
Revised: QD-RL extends Markov Decision

Original: Processes (MDPs) [28] and is formulated as a tuple ( S , U , p, r, γ, m ) .
Revised: Processes (MDPs) [28] and is formulated as a tuple ( S , U , p, r, γ, m ) .

Original: Here, on each timestep ofan episode, an agent observes state s ∈ S and takes action a ∈ U with policy π φ ( a | s ) .
Revised: Here, on each timestep of an episode, an agent observes state s ∈ S and takes action a ∈ U with policy π φ ( a | s ) .

Original: The agenttransitions to the next state s (cid:48) ∈ S according to s (cid:48) ∼ p ( ·| s, a ) and receives reward r ( s, a, s (cid:48) ) .
Revised: The agent transitions to the next state s (cid:48) ∈ S according to s (cid:48) ∼ p ( ·| s, a ) and receives reward r ( s, a, s (cid:48) ) .

Original: Theexpected discounted return is deﬁned as f ( φ ) =
Revised: The expected discounted return is deﬁned as f ( φ ) =

Original: E (cid:104) (cid:80) T t =

Revised: E (cid:104) (cid:80) T t =

Original: γ t r ( s t , a t , s (cid:48) t ) (cid:105) for a discount factorγ ∈ (0 , 1) and episode length T .
Revised: γ t r ( s t , a t , s (cid:48) t ) (cid:105) for a discount factor γ ∈ (0 , 1) and episode length T .

Original: Finally, QD-RL includes a k -dimensional measure function m ( φ )which describes the agent’s behavior during an episode.
Revised: Finally, QD-RL includes a k -dimensional measure function m ( φ ) which describes the agent’s behavior during an episode.

Original:  Background3.1 MAP-Elites
Revised:  Background 3.1 MAP-Elites

Original: Many QD algorithms, including the ones in this paper, are based on the MAP-Elites algorithm.
Revised: Many QD algorithms, including the ones in this paper, are based on the MAP-Elites algorithm.

Original: Theoriginal MAP-Elites algorithm [9, 10] divides the measure space equally along each dimension,resulting in an archive consisting of grid cells.
Revised: The original MAP-Elites algorithm [9, 10] divides the measure space equally along each dimension, resulting in an archive consisting of grid cells.

Original: Initially, MAP-Elites samples solutions from apredeﬁned distribution and inserts them into the archive by mapping each solution to a cell based onits measures.
Revised: Initially, MAP-Elites samples solutions from a predeﬁned distribution and inserts them into the archive by mapping each solution to a cell based on its measures.

Original: On subsequent iterations, MAP-Elites samples a solution φ from the archive, mutates itby adding Gaussian noise (cid:15) ∼ N ( 0 , I ) , and adds the new solution φ (cid:48) to the archive.
Revised: On subsequent iterations, MAP-Elites samples a solution φ from the archive, mutates it by adding Gaussian noise (cid:15) ∼ N ( 0 , I ) , and adds the new solution φ (cid:48) to the archive.

Original: Crucially, if asolution ever lands in the same cell as a previous solution, it is only stored in the archive if it has ahigher objective.
Revised: Crucially, if a solution ever lands in the same cell as a previous solution, it is only stored in the archive if it has a higher objective.

Original: Hence, solutions in the archive are referred to as elites .
Revised: Hence, solutions in the archive are referred to as elites .

Original: Extensions of MAP-Elites include new archive structures [29, 30] and integrations with surrogatemodels [31, 32, 33, 34, 35].
Revised: Extensions of MAP-Elites include new archive structures [29, 30] and integrations with surrogate models [31, 32, 33, 34, 35].

Original: MAP-Elites has also been applied to environment generation forhuman-robot interaction [36, 37] and dataset generation for manipulation [38].
Revised: MAP-Elites has also been applied to environment generation for human-robot interaction [36, 37] and dataset generation for manipulation [38].

Original: Policy Gradient Assisted MAP-Elites (PGA-ME) [13].
Revised: Policy Gradient Assisted MAP-Elites (PGA-ME) [13].

Original: Since MAP-Elites performs poorly whendirectly optimizing modern RL controllers [11, 13, 12], several algorithms [11, 13, 12, 39] havescaled it to such controllers.
Revised: Since MAP-Elites performs poorly when directly optimizing modern RL controllers [11, 13, 12], several algorithms [11, 13, 12, 39] have scaled it to such controllers.

Original: One such algorithm, PGA-ME, replaces the Gaussian noise mutationwith two types of operations: (1) gradient ascent, performed with the TD3 algorithm [40], and(2) crossover, performed with a genetic algorithm [1].
Revised: One such algorithm, PGA-ME, replaces the Gaussian noise mutation with two types of operations: (1) gradient ascent, performed with the TD3 algorithm [40], and (2) crossover, performed with a genetic algorithm [1].

Original: PGA-ME has demonstrated state-of-the-artperformance on the robotic locomotion tasks evaluated in this paper.
Revised: PGA-ME has demonstrated state-of-the-art performance on the robotic locomotion tasks evaluated in this paper.

Original: We include PGA-ME and
Revised: We include PGA-ME and

Original: MAP-Elites as experimental baselines.
Revised: MAP-Elites as experimental baselines.

Original: 3.2 Large-Scale Evolution Strategies (ES)
Revised: 3.2 Large-Scale Evolution Strategies (ES)

Original: ESs [41] optimize continuous parameters by adapting a population of solutions, such that thepopulation is more likely to have high performance.
Revised: ESs [41] optimize continuous parameters by adapting a population of solutions, such that the population is more likely to have high performance.

Original: Here we discuss ESs designed for high- dimensional search spaces.
Revised: Here we discuss ESs designed for highdimensional search spaces.

Original: In Sec.4, we show how these ESs help scale CMA-MAE.
Revised: In Sec.

Original: Covariance Matrix Adaptation Evolution Strategy (CMA-ES) [15].
Revised: Covariance Matrix Adaptation Evolution Strategy (CMA-ES) [15].

Original: CMA-ES is an approximatesecond-order method which follows a natural gradient [42] and demonstrates state-of-the-art results in black-box optimization [43].
Revised: CMA-ES is an approximate second-order method which follows a natural gradient [42] and demonstrates state-of-the-art results in black-box optimization [43].

Original: CMA-ES maintains a Gaussian N ( µ , Σ ) that models the distribution of search directions.
Revised: CMA-ES maintains a Gaussian N ( µ , Σ ) that models the distribution of search directions.

Original: In every iteration, CMA-ES samples λ solutions from the Gaussian and updatesit based on rankings of the solutions’ performance.
Revised: In every iteration, CMA-ES samples λ solutions from the Gaussian and updates it based on rankings of the solutions’ performance.

Original: CMA-ES requires Θ( n 2 ) space and has Θ( n 2 ) runtime per sampled solution due to its n × ncovariance matrix Σ .
Revised: CMA-ES requires Θ( n 2 ) space and has Θ( n 2 ) runtime per sampled solution due to its n × n covariance matrix Σ .

Original: The runtime stems from two operations.
Revised: The runtime stems from two operations.

Original: First, updating Σ requires matrixvector multiplications.
Revised: First, updating Σ requires matrixvector multiplications.

Original: Second, since it is easy to sample from the standard Gaussian N ( 0 , I ) on acomputer, sampling from the Gaussian is implemented as
Revised: Second, since it is easy to sample from the standard Gaussian N ( 0 , I ) on a computer, sampling from the Gaussian is implemented as

Original: [Equation]
Revised: [Equation]

Original: Computing the transformation matrix Σ 12 requires an O ( n 3 ) eigendecomposition, which CMA-ESamortizes to O ( n 2 ) per sampled solution by only recomputing Σ 12 every nλ iterations.
Revised: Computing the transformation matrix Σ 12 requires an O ( n 3 ) eigendecomposition, which CMA-ES amortizes to O ( n 2 ) per sampled solution by only recomputing Σ 12 every nλ iterations.

Original: Nevertheless,this complexity is intractable for high n .
Revised: Nevertheless, this complexity is intractable for high n .

Original: To scale CMA-ES, prior works [18, 19, 20, 21, 22, 23]assume that a full Gaussian is unnecessary for modeling the search distribution in high dimensionsand thus replace the covariance matrix with a more efﬁcient approximation.
Revised: To scale CMA-ES, prior works [18, 19, 20, 21, 22, 23] assume that a full Gaussian is unnecessary for modeling the search distribution in high dimensions and thus replace the covariance matrix with a more efﬁcient approximation.

Original: We discuss two suchapproaches below.
Revised: We discuss two such approaches below.

Original: Separable CMA-ES (sep-CMA-ES) [18].
Revised: Separable CMA-ES (sep-CMA-ES) [18].

Original: As the simplest large-scale CMA-ES variant, sep-CMAES assumes that the problem is separable, i.e., that it may be solved by optimizing each parameterindividually.
Revised: As the simplest large-scale CMA-ES variant, sep-CMAES assumes that the problem is separable, i.e., that it may be solved by optimizing each parameter individually.

Original: Hence, sep-CMA-ES constrains the covariance matrix to be diagonal, so it only storesn parameters.
Revised: Hence, sep-CMA-ES constrains the covariance matrix to be diagonal, so it only stores n parameters.

Original: Updates and sampling then require Θ( n ) runtime per solution, as the matrix-vectormultiplications are replaced with dot products and Σ 12 in Eq.
Revised: Updates and sampling then require Θ( n ) runtime per solution, as the matrix-vector multiplications are replaced with dot products and Σ 12 in Eq.

Original: 1 is computed by taking the square rootof the diagonal elements.
Revised: 1 is computed by taking the square root of the diagonal elements.

Original: sep-CMA-ES outperforms CMA-ES on separable benchmarks and even onsome non-separable problems.
Revised: sep-CMA-ES outperforms CMA-ES on separable benchmarks and even on some non-separable problems.

Original: Limited-Memory Matrix Adaptation ES (LM-MA-ES) [19].
Revised: Limited-Memory Matrix Adaptation ES (LM-MA-ES) [19].

Original: LM-MA-ES is another large-scale
Revised: LM-MA-ES is another large-scale

Original: CMA-ES variant.
Revised: CMA-ES variant.

Original: It maintains a rank- k approximation of the transformation matrix Σ 12 .
Revised: It maintains a rank- k approximation of the transformation matrix Σ 12 .

Original: Thisapproximation consists of k vectors which represent different directions of the search distribution.
Revised: This approximation consists of k vectors which represent different directions of the search distribution.

Original: Since the approximation is rank- k , the space complexity is Θ( kn ) , and the time complexity is Θ( kn )per solution.
Revised: Since the approximation is rank- k , the space complexity is Θ( kn ) , and the time complexity is Θ( kn ) per solution.

Original: Overall, LM-MA-ES performs well on optimization benchmarks, matching otherlarge-scale versions of CMA-ES.
Revised: Overall, LM-MA-ES performs well on optimization benchmarks, matching other large-scale versions of CMA-ES.

Original: OpenAI-ES [14].
Revised: OpenAI-ES [14].

Original: Unlike the algorithms above, OpenAI-ES is not a CMA-ES variant.
Revised: Unlike the algorithms above, OpenAI-ES is not a CMA-ES variant.

Original: Instead,inspired by Natural Evolution Strategies [44], OpenAI-ES maintains an isotropic Gaussian withconstant covariance matrix σ I , and it only updates the mean φ ∗ via gradient ascent with Adam [45].
Revised: Instead, inspired by Natural Evolution Strategies [44], OpenAI-ES maintains an isotropic Gaussian with constant covariance matrix σ I , and it only updates the mean φ ∗ via gradient ascent with Adam [45].

Original: The gradient for this update is computed based on the rank-normalized [46, 44] performances of λsolutions sampled from the Gaussian with mirror sampling [47].
Revised: The gradient for this update is computed based on the rank-normalized [46, 44] performances of λ solutions sampled from the Gaussian with mirror sampling [47].

Original: OpenAI-ES has performed wellacross RL benchmarks [14, 48, 49].
Revised: OpenAI-ES has performed well across RL benchmarks [14, 48, 49].

Original: It has a time complexity of Θ( n ) per sampled solution.
Revised: It has a time complexity of Θ( n ) per sampled solution.

Original: 3.3 Combining ES with MAP-Elites
Revised: 3.3 Combining ES with MAP-Elites

Original: Here we discuss algorithms which enhance MAP-Elites by integrating ESs.
Revised: Here we discuss algorithms which enhance MAP-Elites by integrating ESs.

Original: MAP-Elites with Evolution Strategies (ME-ES) [11].
Revised: MAP-Elites with Evolution Strategies (ME-ES) [11].

Original: ME -ES maintains a solution point φ ∗ andmodiﬁes it with OpenAI-ES in two phases: exploit , where OpenAI-ES optimizes φ ∗ to have highobjective value, and explore , where OpenAI-ES optimizes φ ∗ to be novel [7, 50] (far away fromprevious solutions in measure space).
Revised: ME -ES maintains a solution point φ ∗ and modiﬁes it with OpenAI-ES in two phases: exploit , where OpenAI-ES optimizes φ ∗ to have high objective value, and explore , where OpenAI-ES optimizes φ ∗ to be novel [7, 50] (far away from previous solutions in measure space).

Original: Both phases begin by sampling a new φ ∗ from the archive.
Revised: Both phases begin by sampling a new φ ∗ from the archive.

Original: They then run for k iterations, with only φ ∗ inserted into the archive after each iteration.
Revised: They then run for k iterations, with only φ ∗ inserted into the archive after each iteration.

Original: ME-ES is abaseline in our experiments.
Revised: ME-ES is a baseline in our experiments.

Original: Covariance Matrix Adaptation MAP-Annealing (CMA-MAE) [16].
Revised: Covariance Matrix Adaptation MAP-Annealing (CMA-MAE) [16].

Original: One promising class of QDalgorithms combines CMA-ES [15] with MAP-Elites.
Revised: One promising class of QD algorithms combines CMA-ES [15] with MAP-Elites.

Original: We directly extend the most recent suchalgorithm, CMA-MAE, which has shown state-of-the-art performance in QD benchmark domains.
Revised: We directly extend the most recent such algorithm, CMA-MAE, which has shown state-of-the-art performance in QD benchmark domains.

Original: CMA-MAE maintains an archive which records a threshold t e for each cell e .
Revised: CMA-MAE maintains an archive which records a threshold t e for each cell e .

Original: t e is initialized to auser-deﬁned minimum objective min f .
Revised: t e is initialized to a user-deﬁned minimum objective min f .

Original: When a solution φ is inserted into the archive, it is placedinto its corresponding cell e if its objective value f ( φ ) exceeds t e .
Revised: When a solution φ is inserted into the archive, it is placed into its corresponding cell e if its objective value f ( φ ) exceeds t e .

Original: Then, t e is updated via polyakaveraging, i.e. t e ← (1 − α ) t e + αf ( φ ) , where α ∈ [0 , 1] is the archive learning rate .
Revised: Then, t e is updated via polyak averaging, i.e. t e ← (1 − α ) t e + αf ( φ ) , where α ∈ [0 , 1] is the archive learning rate .

Original: Note that thesolution’s objective value only needs to cross the threshold, rather than the objective value of the elitepreviously in the cell.
Revised: Note that the solution’s objective value only needs to cross the threshold, rather than the objective value of the elite previously in the cell.

Original: Hence, the archive does not always store the best solutions like MAP-Elitesdoes, so implementations must track these best solutions separately.
Revised: Hence, the archive does not always store the best solutions like MAP-Elites does, so implementations must track these best solutions separately.

Original: For generating solutions, CMA-MAE queries a set of emitters on each iteration.
Revised: For generating solutions, CMA-MAE queries a set of emitters on each iteration.

Original: Each emittermaintains its own Gaussian search distribution with mean φ ∗ and covariance matrix Σ .
Revised: Each emitter maintains its own Gaussian search distribution with mean φ ∗ and covariance matrix Σ .

Original: In every iteration, the emitter samples solutions from this Gaussian and inserts the solutions into the archive.
Revised: In every iteration, the emitter samples solutions from this Gaussian and inserts the solutions into the archive.

Original: When a solution φ is inserted, the archive returns an improvement value ∆ i ← f ( φ ) − t e , i.e.,the difference between the objective value of the solution and its corresponding cell’s threshold.
Revised: When a solution φ is inserted, the archive returns an improvement value ∆ i ← f ( φ ) − t e , i.e., the difference between the objective value of the solution and its corresponding cell’s threshold.

Original: The emitter ranks its generated solutions by these improvement values.
Revised: The emitter ranks its generated solutions by these improvement values.

Original: Based on this ranking, theemitter updates φ ∗ and Σ with CMA-ES, such that it is more likely to generate solutions with highimprovement in the future.
Revised: Based on this ranking, the emitter updates φ ∗ and Σ with CMA-ES, such that it is more likely to generate solutions with high improvement in the future.

Original: A key parameter in CMA-MAE is the archive learning rate α .
Revised: A key parameter in CMA-MAE is the archive learning rate α .

Original: When α = 0 , the threshold alwaysremains at min f , and hence the improvement is always equivalent to the objective value (minusthe constant min f ).
Revised: When α = 0 , the threshold always remains at min f , and hence the improvement is always equivalent to the objective value (minus the constant min f ).

Original: This makes CMA-MAE equivalent to CMA-ES, as it optimizes solely for theobjective function.
Revised: This makes CMA-MAE equivalent to CMA-ES, as it optimizes solely for the objective function.

Original: However, when α = 1 , the threshold is always equivalent to the objective valueof the elite currently in the cell, which means there is minimal improvement for inserting a solutioninto that cell.
Revised: However, when α = 1 , the threshold is always equivalent to the objective value of the elite currently in the cell, which means there is minimal improvement for inserting a solution into that cell.

Original: In this case, CMA-MAE is equivalent to the previously developed CMA-ME [17],where discovering new cells in the archive (i.e., exploration) always takes priority over improvingthe objective value of cells where there is already a solution.
Revised: In this case, CMA-MAE is equivalent to the previously developed CMA-ME [17], where discovering new cells in the archive (i.e., exploration) always takes priority over improving the objective value of cells where there is already a solution.

Original: Hence, by varying α from 0 to 1, wetrade off between optimizing the objective and moving away from previously discovered elites.
Revised: Hence, by varying α from 0 to 1, we trade off between optimizing the objective and moving away from previously discovered elites.

Original: Inparticular, we use a low α value due to the need for optimization in robotic locomotion domains.
Revised: In particular, we use a low α value due to the need for optimization in robotic locomotion domains.

Original: Covariance Matrix Adaptation MAP-Elites via a Gradient Arborescence (CMA-MEGA) [27].
Revised: Covariance Matrix Adaptation MAP-Elites via a Gradient Arborescence (CMA-MEGA) [27].

Original: In the differentiable quality diversity (DQD) setting, where exact objective and measure gradientsare available, CMA-ME and gradient ascent have been combined to form the state-of-the-art CMAMEGA algorithm.
Revised: In the differentiable quality diversity (DQD) setting, where exact objective and measure gradients are available, CMA-ME and gradient ascent have been combined to form the state-of-the-art CMAMEGA algorithm.

Original: To extend CMA-MEGA to QD-RL, prior work replaces the exact gradientswhich CMA-MEGA relies on with gradient approximations, since the RL environment is usuallynon-differentiable.
Revised: To extend CMA-MEGA to QD-RL, prior work replaces the exact gradients which CMA-MEGA relies on with gradient approximations, since the RL environment is usually non-differentiable.

Original: This results in two variants: CMA-MEGA (ES) approximates all gradients with
Revised: This results in two variants: CMA-MEGA (ES) approximates all gradients with

Original: OpenAI-ES [14], while CMA-MEGA (TD3, ES) approximates the objective gradient with TD3 [40]and measure gradients with OpenAI-ES.
Revised: OpenAI-ES [14], while CMA-MEGA (TD3, ES) approximates the objective gradient with TD3 [40] and measure gradients with OpenAI-ES.

Original: These variants are baselines in our experiments.
Revised: These variants are baselines in our experiments.

Original:  Scaling Covariance Matrix Adaptation MAP-Annealing
Revised:  Scaling Covariance Matrix Adaptation MAP-Annealing

Original: In CMA-MAE, each emitter relies on CMA-ES to update the Gaussian search distribution from whichit samples solutions.
Revised: In CMA-MAE, each emitter relies on CMA-ES to update the Gaussian search distribution from which it samples solutions.

Original: Since CMA-ES requires Θ( n 2 ) runtime per solution (where n is the solutiondimension), CMA-MAE is intractable for neural network controllers, where there are thousands tomillions of parameters.
Revised: Since CMA-ES requires Θ( n 2 ) runtime per solution (where n is the solution dimension), CMA-MAE is intractable for neural network controllers, where there are thousands to millions of parameters.

Original: To scale CMA-MAE, we propose three variants which substitute CMA-ESwith large-scale ESs.
Revised: To scale CMA-MAE, we propose three variants which substitute CMA-ES with large-scale ESs.

Original: These variants differ primarily in the complexity of their covariance matrixapproximation:
Revised: These variants differ primarily in the complexity of their covariance matrix approximation:

Original: • sep-CMA-MAE substitutes sep-CMA-ES [18], which constrains the covariance matrix Σto be diagonal, resulting in Θ( n ) complexity.
Revised: • sep-CMA-MAE substitutes sep-CMA-ES [18], which constrains the covariance matrix Σ to be diagonal, resulting in Θ( n ) complexity.

Original: • LM-MA-MAE substitutes LM-MA-ES [19], which approximates the transformation matrixΣ 12 with k (cid:28) n n -dimensional vectors, leading to Θ( kn ) complexity.• OpenAI-MAE substitutes OpenAI-ES [14], which sets the covariance matrix Σ to be theidentity matrix.
Revised: • LM-MA-MAE substitutes LM-MA-ES [19], which approximates the transformation matrix Σ 12 with k (cid:28) n n -dimensional vectors, leading to Θ( kn ) complexity. • OpenAI-MAE substitutes OpenAI-ES [14], which sets the covariance matrix Σ to be the identity matrix.

Original: Though Σ is constant, various operations still necessitate Θ( n ) complexity.
Revised: Though Σ is constant, various operations still necessitate Θ( n ) complexity.

Original: The listed complexities refer to (1) the space required per emitter, as each emitter maintains its own
Revised: The listed complexities refer to (1) the space required per emitter, as each emitter maintains its own

Original: ES instance, and (2) the runtime required per  solution, which is the same regardless of the number ofemitters.
Revised: ES instance, and (2) the runtime required per sampled solution, which is the same regardless of the number of emitters.

Original: Algorithm 1 shows the pseudocode for these CMA-MAE variants, and Fig.
Revised: Algorithm 1 shows the pseudocode for these CMA-MAE variants, and Fig.

Original: 1 shows an overview ofthe variants.
Revised: 1 shows an overview of the variants.

Original: Each variant proceeds as follows.
Revised: Each variant proceeds as follows.

Original: First, it initializes the archive along with each emitter’s
Revised: First, it initializes the archive along with each emitter’s

Original: ES parameters (line 2-3).
Revised: ES parameters (line 2-3).

Original: This step includes initializing the covariance matrix approximation ˜ Σ inlieu of CMA-MAE’s full covariance matrix Σ .
Revised: This step includes initializing the covariance matrix approximation ˜ Σ in lieu of CMA-MAE’s full covariance matrix Σ .

Original: Next, the variant repeatedly queries the emitters forsolutions (line 4).
Revised: Next, the variant repeatedly queries the emitters for solutions (line 4).

Original: Each emitter (line 5) samples λ solutions from the distribution N ( φ ∗ , ˜ Σ ) (line 6-7).
Revised: Each emitter (line 5) samples λ solutions from the distribution N ( φ ∗ , ˜ Σ ) (line 6-7).

Original: Note that the sampling procedure depends on the approximation employed by the variant.
Revised: Note that the sampling procedure depends on the approximation employed by the variant.

Original: Oncesampled, the solutions are evaluated and inserted into the archive if they cross their cell’s
Revised: Once sampled, the solutions are evaluated and inserted into the archive if they cross their cell’s

Original: thresholdt e (line 8-15).
Revised: threshold t e (line 8-15).

Original: Then, the ES’s parameters (including ˜ Σ ) are updated based on the improvementranking of the solutions (line 16-17), such that the emitter is more likely to sample solutions with highimprovement on the next iteration.
Revised: Then, the ES’s parameters (including ˜ Σ ) are updated based on the improvement ranking of the solutions (line 16-17), such that the emitter is more likely to sample solutions with high improvement on the next iteration.

Original: Finally, the emitter is restarted if the ES converges (line 18-20).
Revised: Finally, the emitter is restarted if the ES converges (line 18-20).

Original: We adopt the default update and convergence rules from each ES.
Revised: We adopt the default update and convergence rules from each ES.

Original:  Experiments
Revised:  Experiments

Original: To test our CMA-MAE variants’ ability to create diverse, high-performing locomotion controllers weevaluate it on four robotic locomotion tasks from the QDGym [24] benchmark based on PyBullet [51,52] and OpenAI Gym [53].
Revised: To test our CMA-MAE variants’ ability to create diverse, high-performing locomotion controllers, we evaluate it on four robotic locomotion tasks from the QDGym [24] benchmark based on PyBullet [51, 52] and OpenAI Gym [53].

Original: In each environment (Fig. 2), the objective is to walk forward quickly,while the measures track the proportion of time in an episode that each of the robot’s feet touches theground, i.e., if a robot has four legs, it has four measures.
Revised: In each environment (Fig. 2), the objective is to walk forward quickly, while the measures track the proportion of time in an episode that each of the robot’s feet touches the ground, i.e., if a robot has four legs, it has four measures.

Original: As prior work [12] notes, these environments are challenging because they require focusing onobjective optimization.
Revised: As prior work [12] notes, these environments are challenging because they require focusing on objective optimization.

Original: It is easy to ﬁll the archive with poor controllers that achieve differentmeasures by standing still and lifting the feet up and down, but making these controllers walk forwardis much more difﬁcult.
Revised: It is easy to ﬁll the archive with poor controllers that achieve different measures by standing still and lifting the feet up and down, but making these controllers walk forward is much more difﬁcult.

Original: We compare our variants with ﬁve baselines: PGA-ME, two CMA-MEGAvariants (CMA-MEGA (ES) and CMA-MEGA (TD3, ES)), ME-ES, and MAP-Elites.
Revised: We compare our variants with ﬁve baselines: PGA-ME, two CMA-MEGA variants (CMA-MEGA (ES) and CMA-MEGA (TD3, ES)), ME-ES, and MAP-Elites.

Original: 5.1 Hyperparameters
Revised: 5.1 Hyperparameters

Original: As in prior work [13, 12], the robot controller is a neural network mapping states to actions.
Revised: As in prior work [13, 12], the robot controller is a neural network mapping states to actions.

Original: Thenetwork has two hidden layers of size 128 and tanh activations on every layer but the input, and it isinitialized with Xavier initialization.
Revised: The network has two hidden layers of size 128 and tanh activations on every layer but the input, and it is initialized with Xavier initialization.

Original: The number of parameters in each domain is as follows: QD
Revised: The number of parameters in each domain is as follows: QD

Original: Ant (21,256), QD Half-Cheetah (20,742), QD Hopper (18,947), QD Walker (20,230).
Revised: Ant (21,256), QD Half-Cheetah (20,742), QD Hopper (18,947), QD Walker (20,230).

Original: [Figure]
Revised: [Figure]

Original: All algorithms are allocated 1 million evaluations, with each solution evaluated for one episode.
Revised: All algorithms are allocated 1 million evaluations, with each solution evaluated for one episode.

Original: PGA-ME, CMA-MEGA (ES), CMA-MEGA (TD3, ES), and ME-ES use the same hyperparametersas in their prior works [13, 12, 11], except ME-ES uses 200 samples.
Revised: PGA-ME, CMA-MEGA (ES), CMA-MEGA (TD3, ES), and ME-ES use the same hyperparameters as in their prior works [13, 12, 11], except ME-ES uses 200 samples.

Original: MAP-Elites uses a noisestandard deviation of σ = 0 .
Revised: MAP-Elites uses a noise standard deviation of σ = 0 .

Original: 02 and a batch size of 100.
Revised: 02 and a batch size of 100.

Original: All CMA-MAE variants run with a low archive learning rate α = 0 .
Revised: All CMA-MAE variants run with a low archive learning rate α = 0 .

Original: 001 to encourage optimization ofthe objective and with ψ = 5 emitters, each with batch size λ =40 and initial step size σ = 0 .
Revised: 001 to encourage optimization of the objective and with ψ = 5 emitters, each with batch size λ = and initial step size σ = 0 .

Original: LM-MA-MAE sets k = λ
Revised: LM-MA-MAE sets k = λ

Original: = 40 .
Revised: = 40 .

Original: The Adam optimizer for OpenAI-ES in OpenAI-MAE useslearning rate 0.01 and L2 regularization coefﬁcient 0.005.
Revised: The Adam optimizer for OpenAI-ES in OpenAI-MAE uses learning rate 0.01 and L2 regularization coefﬁcient 0.005.

Original: For min f , QDGym does not have predeﬁned minimum objectives, but we adopt values from priorwork [12] which recorded the minimum objective inserted into an archive during their experiments:
Revised: For min f , QDGym does not have predeﬁned minimum objectives, but we adopt values from prior work [12] which recorded the minimum objective inserted into an archive during their experiments:

Original: QD Ant (-374.70), QD Half-Cheetah (-2,797.52), QD Hopper (-362.09), QD Walker (-67.17).
Revised: QD Ant (-374.70), QD Half-Cheetah (-2,797.52), QD Hopper (-362.09), QD Walker (-67.17).

Original: 5.2 Experimental Design
Revised: 5.2 Experimental Design

Original: We conduct a between-groups study with the algorithm (LM-MA-MAE, sep-CMA-MAE, OpenAIMAE , PGA-ME, CMA-MEGA (ES), CMA-MEGA (TD3, ES), ME-ES, MAP-Elites) and environment (QD Ant, QD Half-Cheetah, QD Hopper, QD Walker) as independent variables and QD score asdependent variable.
Revised: We conduct a between-groups study with the algorithm (LM-MA-MAE, sep-CMA-MAE, OpenAIMAE, PGA-ME, CMA-MEGA (ES), CMA-MEGA (TD3, ES), ME-ES, MAP-Elites) and environment (QD Ant, QD Half-Cheetah, QD Hopper, QD Walker) as independent variables and QD score as dependent variable.

Original: The QD score [6] measures performance holistically by summing the objectivesof all elites in the archive.
Revised: The QD score [6] measures performance holistically by summing the objectives of all elites in the archive.

Original: To ensure that no elite subtracts from the QD score (objectives may benegative), we subtract min f from all elites’ objectives before computing the QD score.
Revised: To ensure that no elite subtracts from the QD score (objectives may be negative), we subtract min f from all elites’ objectives before computing the QD score.

Original: We repeatour experiments 5 times to test three hypotheses:
Revised: We repeat our experiments 5 times to test three hypotheses:

Original: H1:
Revised: H1:

Original: All CMA-MAE variants will outperform prior ES-based methods, i e., ME-ES and CMA-MEGA(ES).H2: All CMA-MAE variants will outperform deep RL-based methods,
Revised: All CMA-MAE variants will outperform prior ES-based methods, i.e., ME-ES and CMA-MEGA (ES). H2: All CMA-MAE variants will outperform deep RL-based methods,

Original: i
Revised: i

Original: e., PGA-ME and CMA-MEGA(TD3, ES).H3: The performances of the CMA-MAE variants will be ordered as LM-MA-MAE > sep-CMA-MAE> OpenAI-MAE.
Revised: e., PGA-ME and CMA-MEGA (TD3, ES). H3: The performances of the CMA-MAE variants will be ordered as LM-MA-MAE > sep-CMA-MAE > OpenAI-MAE.

Original: H1 and H2 are based on the key difference between the variants and the baselines, i.e., the smoothimprovement ranking based on archive thresholds.
Revised: H1 and H2 are based on the key difference between the variants and the baselines, i.e., the smooth improvement ranking based on archive thresholds.

Original: Since CMA-MEGA (ES) and CMA-MEGA(TD3, ES) use a standard MAP-Elites archive (equivalent to setting α = 1 in CMA-MAE), we thinkthey will focus too much on exploration and be unable to optimize existing solutions.
Revised: Since CMA-MEGA (ES) and CMA-MEGA (TD3, ES) use a standard MAP-Elites archive (equivalent to setting α = 1 in CMA-MAE), we think they will focus too much on exploration and be unable to optimize existing solutions.

Original: Meanwhile,
Revised: Meanwhile,

Original: PGA-ME and ME-ES separate measure space exploration from objective optimization; PGA-ME hastwo types of operations, while ME-ES has two distinct phases.
Revised: PGA-ME and ME-ES separate measure space exploration from objective optimization; PGA-ME has two types of operations, while ME-ES has two distinct phases.

Original: We hypothesize that this separationwill not work as well as blending the two aspects together via the improvement ranking.
Revised: We hypothesize that this separation will not work as well as blending the two aspects together via the improvement ranking.

Original: The ordering in H3 is based on the complexity of each variant’s covariance matrix approximation.
Revised: The ordering in H3 is based on the complexity of each variant’s covariance matrix approximation.

Original: Essentially, given the difﬁculty of the locomotion domains, we hypothesize that a better representationof the search distribution will lead to higher performance.
Revised: Essentially, given the difﬁculty of the locomotion domains, we hypothesize that a better representation of the search distribution will lead to higher performance.

Original: 5.3 Implementation
Revised: 5.3 Implementation

Original: Each algorithm runs with 100 CPUs on a high-performance cluster.
Revised: Each algorithm runs with 100 CPUs on a high-performance cluster.

Original: Algorithms which train TD3 areallocated one NVIDIA Tesla P100 GPU.
Revised: Algorithms which train TD3 are allocated one NVIDIA Tesla P100 GPU.

Original: We obtained data for all baselines with consent from theauthors of prior work [12].
Revised: We obtained data for all baselines with consent from the authors of prior work [12].

Original: All algorithms are implemented with the pyribs library [55].
Revised: All algorithms are implemented with the pyribs library [55].

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original:  Results
Revised:  Results

Original: Fig.
Revised: Fig.

Original: 3 summarizes our results.
Revised: 3 summarizes our results.

Original: To test our hypotheses (Sec. 5.2), we ﬁrst ran a two-way ANOVA toexamine the effect of the algorithm and environment on QD score.
Revised: To test our hypotheses (Sec. 5.2), we ﬁrst ran a two-way ANOVA to examine the effect of the algorithm and environment on QD score.

Original: The ANOVA showed a signiﬁcantinteraction between algorithm and environment ( F (21 , 128) = 18 .
Revised: The ANOVA showed a signiﬁcant interaction between algorithm and environment ( F (21 , 128) = 18 .

Original: 01 , p < 0 .
Revised: 01 , p < 0 .

Original: Thus, for eachhypothesis, we report the results of simple main effects analysis with Bonferroni corrections.
Revised: Thus, for each hypothesis, we report the results of simple main effects analysis with Bonferroni corrections.

Original: H1:
Revised: H1:

Original: All CMA-MAE variants outperformed ME-ES in all environments and CMA-MEGA (ES) intwo environments (QD Hopper and QD Walker).
Revised: All CMA-MAE variants outperformed ME-ES in all environments and CMA-MEGA (ES) in two environments (QD Hopper and QD Walker).

Original: In QD Ant and QD Half-Cheetah, there was nosigniﬁcant difference between the variants and CMA-MEGA (ES).
Revised: In QD Ant and QD Half-Cheetah, there was no signiﬁcant difference between the variants and CMA-MEGA (ES).

Original: H2:
Revised: H2:

Original: In QD Ant, Half-Cheetah, and Hopper, the variants had no signiﬁcant difference with PGA-MEand CMA-MEGA (TD3, ES), except OpenAI-MAE was signiﬁcantly worse than PGA-ME in QDAnt.
Revised: In QD Ant, Half-Cheetah, and Hopper, the variants had no signiﬁcant difference with PGA-ME and CMA-MEGA (TD3, ES), except OpenAI-MAE was signiﬁcantly worse than PGA-ME in QD Ant.

Original: In QD Walker, PGA-ME and CMA-MEGA (TD , ES) outperformed all variants.H3:
Revised: In QD Walker, PGA-ME and CMA-MEGA (TD3, ES) outperformed all variants.

Original: We ran pairwise comparisons between all three pairs of variants. In QD Ant, there wasno signiﬁcant difference between LM-MA-MAE and sep-CMA-MAE, but OpenAI-MAE was signiﬁcantly worse than LM-MA-MAE. In QD Half-Cheetah, sep-CMA-MAE outperformed both OpenAI-MAE and LM-MA-MAE, but there was no signiﬁcant difference between OpenAI-MAE and LM-MA-MAE.
Revised: and LM-MA-MAE.

Original: In QD Hopper, sep-CMA-MAE outperformed OpenAI-MAE, but there wasno signiﬁcant difference between sep-CMA-MAE and LM-MA-MAE or between LM-MA-MAEand OpenAI-MAE.
Revised: In QD Hopper, sep-CMA-MAE outperformed OpenAI-MAE, but there was no signiﬁcant difference between sep-CMA-MAE and LM-MA-MAE or between LM-MA-MAE and OpenAI-MAE.

Original: Finally, there was no signiﬁcant difference between any pair of variants in QD
Revised: Finally, there was no signiﬁcant difference between any pair of variants in QD

Original: Walker.
Revised: Walker.

Original:  Discussion and Conclusion
Revised:  Discussion and Conclusion

Original: Our results show that ES-based methods are a viable alternative in quality diversity for reinforcementlearning (QD-RL).
Revised: Our results show that ES-based methods are a viable alternative in quality diversity for reinforcement learning (QD-RL).

Original: First, the analysis of H1 shows that the CMA-MAE variants are the highestperforming ES-based method in QD-RL, either matching or exceeding the existing CMA-MEGA(ES) and ME-ES on all domains.
Revised: First, the analysis of H1 shows that the CMA-MAE variants are the highestperforming ES-based method in QD-RL, either matching or exceeding the existing CMA-MEGA (ES) and ME-ES on all domains.

Original: Additionally, the analysis of H2 shows that the variants arecomparable to RL-based methods (including the state-of-the-art PGA-ME) in 3 of 4 tasks.
Revised: Additionally, the analysis of H2 shows that the variants are comparable to RL-based methods (including the state-of-the-art PGA-ME) in 3 of 4 tasks.

Original: While thevariants do not solve QD Walker, prior work [12] highlights the importance of deep RL in this task,as only algorithms with TD3 (PGA-ME and CMA-MEGA (TD3, ES)) have performed well here.
Revised: While the variants do not solve QD Walker, prior work [12] highlights the importance of deep RL in this task, as only algorithms with TD3 (PGA-ME and CMA-MEGA (TD3, ES)) have performed well here.

Original: Overall, these results show that the improvement ranking and annealing mechanisms of CMA-MAEenable it to focus on the optimization of the objective, which is necessary in the robotic locomotiontasks.
Revised: Overall, these results show that the improvement ranking and annealing mechanisms of CMA-MAE enable it to focus on the optimization of the objective, which is necessary in the robotic locomotion tasks.

Original: We note that a beneﬁt of ES-based methods is their generality.
Revised: We note that a beneﬁt of ES-based methods is their generality.

Original: While PGA-ME performs well onthe robotic locomotion tasks, it is designed only for QD-RL settings, since its TD3 component isapplicable only in MDP domains.
Revised: While PGA-ME performs well on the robotic locomotion tasks, it is designed only for QD-RL settings, since its TD3 component is applicable only in MDP domains.

Original: On the other hand, the CMA-MAE variants treat the objective as ablack box and can be applied to general QD problems outside of robotics.
Revised: On the other hand, the CMA-MAE variants treat the objective as a black box and can be applied to general QD problems outside of robotics.

Original: Furthermore, ES-based methods have attractive practical beneﬁts.
Revised: Furthermore, ES-based methods have attractive practical beneﬁts.

Original: Speciﬁcally:
Revised: Speciﬁcally:

Original: (1) The CMA-MAE variants are noticeably light on computation.
Revised: (1) The CMA-MAE variants are noticeably light on computation.

Original: While PGA-ME and CMA-MEGA(TD3, ES) both train a TD3 critic, a lengthy process which requires a GPU, the CMA-MAE variantsrun entirely on CPU and are three times faster (7 hrs vs 20 hrs in QD Ant and QD Half-Cheetah, and4 hrs vs 13 hrs in QD Hopper and QD Walker).
Revised: While PGA-ME and CMA-MEGA (TD3, ES) both train a TD3 critic, a lengthy process which requires a GPU, the CMA-MAE variants run entirely on CPU and are three times faster (7 hrs vs 20 hrs in QD Ant and QD Half-Cheetah and 4 hrs vs 13 hrs in QD Hopper and QD Walker).

Original: (2) The CMA-MAE variants have substantially fewer hyperparameters than PGA-ME and the CMAMEGA variants since they depend on CMA-ES and its variants, which are designed to be parameter- ized only by an initial step size σ and batch size λ .
Revised: The CMA-MAE variants have substantially fewer hyperparameters than PGA-ME and the CMAMEGA variants since they depend on CMA-ES and its variants, which are designed to be parameterized only by an initial step size σ and batch size λ .

Original: Hence, the CMA-MAE variants only require 5hyperparameters ( ψ, λ, σ, α, min f , see Algorithm 1).
Revised: Hence, the CMA-MAE variants only requirehyperparameters ( ψ, λ, σ, α, min f , see Algorithm 1).

Original: In contrast, PGA-ME and CMA-MEGA (TD3,ES) require a large number of parameters for TD3 and their other components.
Revised: In contrast, PGA-ME and CMA-MEGA (TD3, ES) require a large number of parameters for TD3 and their other components.

Original: When comparing the CMA-MAE variants in H3 , we observe that OpenAI-MAE performs significantly worse than one other variant in three out of four tasks.
Revised: When comparing the CMA-MAE variants in H3 , we observe that OpenAI-MAE performs significantly worse than one other variant in three out of four tasks.

Original: This is because OpenAI-MAEapproximates the covariance matrix with a constant identity matrix, as opposed to the more complexapproximations of sep-CMA-MAE and LM-MA-MAE.
Revised: This is because OpenAI-MAE approximates the covariance matrix with a constant identity matrix, as opposed to the more complex approximations of sep-CMA-MAE and LM-MA-MAE.

Original: On the other hand, there does not seem tobe a signiﬁcant difference between sep-CMA-MAE and LM-MA-MAE.
Revised: On the other hand, there does not seem to be a signiﬁcant difference between sep-CMA-MAE and LM-MA-MAE.

Original: This result warrants furtherinvestigation, and it indicates that both the rank- k and diagonal approximations of the covariancematrix are sufﬁcient for modeling the distribution of search directions in the locomotion tasks.
Revised: This result warrants further investigation, and it indicates that both the rank- k and diagonal approximations of the covariance matrix are sufﬁcient for modeling the distribution of search directions in the locomotion tasks.

Original: In conclusion, our work scales CMA-MAE to robotic locomotion controllers by replacing the CMAES component of CMA-MAE with efﬁcient approximations.
Revised: In conclusion, our work scales CMA-MAE to robotic locomotion controllers by replacing the CMAES component of CMA-MAE with efﬁcient approximations.

Original: Our results show the CMA-MAEvariants are effective in benchmark locomotion tasks and have fewer hyperparameters than thedeep RL baselines, making them attractive to practitioners.
Revised: Our results show the CMA-MAE variants are effective in benchmark locomotion tasks and have fewer hyperparameters than the deep RL baselines, making them attractive to practitioners.

Original: Since CMA-MAE is a general purposealgorithm, we are excited about future work that will test the CMA-MAE variants in domains beyondrobot locomotion, such as robotic manipulation [25] and scenario generation [26].
Revised: Since CMA-MAE is a general purpose algorithm, we are excited about future work that will test the CMA-MAE variants in domains beyond robot locomotion, such as robotic manipulation [25] and scenario generation [26].

Original: Future work willalso test the pretrained controllers in the real world and will explore the computational beneﬁts ofrecent hardware-accelerated frameworks [56]
Revised: Future work will also test the pretrained controllers in the real world and will explore the computational beneﬁts of recent hardware-accelerated frameworks [56].

Revised: Acknowledgments and Disclosure of Funding

Revised: This work was partially supported by the NSF CAREER (#