Original: Generative Evolutionary Strategy For Black-Box Optimizations
Revised: Generative Evolutionary Strategy For Black-Box Optimizations

Original: Abstract
Revised: Abstract

Original: Many scientific and technological problems are related to optimization.
Revised: Many scientific and technological problems are related to optimization.

Original: Amongthem, black-box optimization in high-dimensional space is particularly challenging.
Revised: Amongthem, black-box optimization in high-dimensional space is particularly challenging.

Original: Recent neural network-based black-box optimization studies have shown noteworthy achievements.
Revised: Recent neural network-based black-box optimization studies have shown noteworthy achievements.

Original: However, their capability in high-dimensional search space isstill limited.
Revised: However, their capability in high-dimensional search space isstill limited.

Original: This study proposes a black-box optimization method based on evolution strategy and generative neural network model.
Revised: This study proposes a black-box optimization method based on evolution strategy and generative neural network model.

Original: We designed the algorithm sothat the evolutionary strategy and the generative neural network model work cooperatively with each other.
Revised: We designed the algorithm sothat the evolutionary strategy and the generative neural network model work cooperatively with each other.

Original: This hybrid model enables reliable training of surrogatenetworks; it optimizes multi-objective, high-dimensional, and stochastic blackbox functions.
Revised: This hybrid model enables reliable training of surrogatenetworks; it optimizes multi-objective, high-dimensional, and stochastic blackbox functions.

Original: In this experiment, our method outperforms baseline optimizationmethods, including evolution strategies, and a Bayesian optimization
Revised: In this experiment, our method outperforms baseline optimizationmethods, including evolution strategies, and a Bayesian optimization

Original:  Introduction
Revised:  Introduction

Original: Optimization is one of the most crucial issues in science and technology.
Revised: Optimization is one of the most crucial issues in science and technology.

Original: Various simulations andexperiments work as black-box functions, and there have also been innumerable optimization studies.
Revised: Various simulations andexperiments work as black-box functions, and there have also been innumerable optimization studies.

Original: Gradient-based optimization methods are easy choices for differentiable functions or simple convexfunctions.
Revised: Gradient-based optimization methods are easy choices for differentiable functions or simple convexfunctions.

Original: However, black-box functions are often non-differentiable and non-convex.
Revised: However, black-box functions are often non-differentiable and non-convex.

Original: Furthermore,they can be multi-objective and stochastic.
Revised: Furthermore,they can be multi-objective and stochastic.

Original: A simple description of multi-objective black-boxoptimization is as follows
Revised: A simple description of multi-objective black-boxoptimization is as follows

Original: Optimize ( 
Revised: Optimize ( f

Revised: [Equation]

Revised: [Equation]

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: The search space is defined in real space R , where N is the dimension.
Revised: The search space is defined in real space R , where N is the dimension.

Original: f i is a single-objectivefunction, which can be stochastic.
Revised: f i is a single-objectivefunction, which can be stochastic.

Original: The evaluation of electronic device design is a practical problem of the  optimization.
Revised: The evaluation of electronic device design is a practical problem of the black-box optimization.

Original: Since thedevice simulators have time-sequential input-output structures, it seems like they can be solved inreinforcement learning.
Revised: Since the device simulators have time-sequential input-output structures, it seems like they can besolved in reinforcement learning.

Original: However, if the cost of simulator is too high, it is almost impossible to observetime-sequential data.
Revised: However, if the cost of simulator is too high, it will be almostimpossible to observe time-sequential data.

Original: Therefore, the evaluation problem is defined as a black-box optimization, inthis case.
Revised: Therefore, the evaluation problem is defined as a blackbox optimization, in this case.

Original: For practical purposes, researchers have studied optimization methods in various ways.
Revised: For practical purposes, researchers have studied optimization methods in various ways.

Original: Typically, Bayesian optimization [1][2][3] and evolutionary strategies [7]-[20] are widely used.
Revised: Typically, Bayesian optimization [1][2][3] and evolutionary strategies [7]-[20] are widely used.

Original: Notably, the Bayesian optimization is advantageous when the cost of the target function is high and the number of function calls is limited.
Revised: Notably, the Bayesian optimization is advantageous when the cost of the target function is high and the number of function calls is limited.

Original: The estimating process of Bayesian optimization is very efficient when the number of search points is small.
Revised: The estimating process of Bayesian optimization is very efficient when the number of search points is small.

Original: On the contrary, it becomes inefficient when the number of search points increases.
Revised: On the contrary, it becomes inefficient when the number of search points increases.

Original: Therefore,
Revised: Therefore,

Original: Evolution Strategy
Revised: Evolution Strategy

Original: Generative N.N. model
Revised: Generative N.N. model

Original: Stabletraining
Revised: Stable

Original: [Figure]
Revised: [Figure]

Original: Bayesian optimization may not be appropriate in high-dimensional problems that require a largenumber of function calls.
Revised: Bayesian optimization may not be appropriate in high-dimensional problems that require a largenumber of function calls.

Original: Meanwhile, evolution strategies can be better choices for aforementioned cases.
Revised: Meanwhile, evolution strategies can be better choices for aforementioned cases.

Original: In most evolutionstrategies, the computational cost does not drastically increase according to the number of searchpoints.
Revised: In most evolutionstrategies, the computational cost does not drastically increase according to the number of searchpoints.

Original: Nevertheless, it does not mean that they can avoid the curse of dimensionality .
Revised: Nevertheless, it does not mean that they can avoid the curse of dimensionality .

Original: The optimization performance of the evolutionary strategies also decreases rapidly as the dimension of theblack-box increases.
Revised: The optimization performance of the evolutionary strategies also decreases rapidly as the dimension of theblack-box increases.

Original: Although SEP-CMA[55], VD-CMA[56], and LM-CMA[57] have shown optimization capability in high-dimensional space of convex functions, optimization of high-dimensionalnon-convex problems still seems difficult.
Revised: Although SEP-CMA[55], VD-CMA[56], and LM-CMA[57] have shown optimization capability in high-dimensional space of convex functions, optimization of high-dimensionalnon-convex problems still seems difficult.

Original: Generative neural network-based models are recent approaches.
Revised: Generative neural network-based models are recent approaches.

Original: They show noteworthy performancein test function optimizations, but their capability seems to be limited to single-objective functionsand to 100-dimension [21][28].
Revised: They show noteworthy performancein test function optimizations, but their capability seems to be limited to single-objective functionsand to 100-dimension [21][28].

Original: We present GEO: Generative Evolutionary Optimization, a methodfor general black-box optimizations.
Revised: We present GEO: Generative Evolutionary Optimization, a methodfor general black-box optimizations.

Original: It is designed to optimize stochastic, multi-objective, andhigh-dimensional black-box problems.
Revised: It is designed to optimize stochastic, multi-objective, and highdimensional black-box problems.

Original: We show that GEO outperforms baseline methods in finding Pareto fronts of Styblinski-Tang [42], Ackley [39], Rastrigin [36][37][38], Rosenbrock [40][41], 
Revised: We show that GEO outperforms baseline methods in finding Paretofronts of Styblinski-Tang [42], Ackley [39], Rastrigin [36][37][38], Rosenbrock [40][41], ZDT1,

Original: ZDT1, ZDT2, and ZDT3 [43] test functions.
Revised: ZDT2, and ZDT3 [43] test functions.

Original: Also, by converting Cartpole-V1 [44](and LeNet-5[45]) to high-dimensional black-box problems, we show that GEO can generate optimized sequences(and images) without observing intermediate states (and without the direct gradient flow from targetfunctions).
Revised: Also, by converting Cartpole-V1 [44] to high-dimensionalblack-box problems, we show that GEO can be used in sequential problems. We also tested LeNet-5[45] to see how it generates sub-manifold structures.

Original:  Related works
Revised:  Related works

Original: GEO is related to Evolutionary Generative Adversarial Networks (EGAN) [22]. Like Local Generative
Revised: GEO is related to Evolutionary Generative Adversarial Networks (EGAN) [22], and Local Generative

Original: Surrogates Optimization (L-GSO) [21], surrogate model-based optimizations are also related to GEO.
Revised: Surrogates Optimization (L-GSO) [21].

Original: This section briefly introducesthe network-based black-box optimizations and EGAN to show how GEO can adapt them.
Revised: This section briefly introduces them.

Original: 2.1 L-GSO
Revised: 2.1 L-GSO

Original: The black-box optimization with gradient descent method is a tricky task since the black-box doesnot provide gradient information directly. The L-GSO study shows that alocal surrogate modelcan be a good solution for it.
Revised: L-GSO is a surrogate network model based black-box optimizer. It has a surrogate network and agenerator network.

Original: In this study, the surrogate network  estimates a local shape oftheobjective function, and the locality is a crucial idea of L-GSO.
Revised: The main idea of L-GSO is that the surrogate network only estimates a localshape of the objective function.

Original: By using local sampling of ε -cube (orball) around a currently-optimized point, a stable surrogate model provides local gradients. Anotheradvantage ofL-GSO is that theneural network-based surrogate model naturally works in stochastic environments. Because surrogatenetworks estimate a distribution probabilistically, the model cansuggest gradients even ina stochastic function.
Revised: Since the stabilization of the surrogate network is difficult, theysuggest only to surrogate a local region. Also, the optimizer can be used in stochastic environmentssince it works in the neural network.

Original: It is shown that L-GSO outperforms baselineoptimizers (Bayesian optimization and CMA-ES [9][10] in dimension = 10 and sub-manifolddimension = 100 problems).
Revised: It is shown that L-GSO outperforms baseline optimizers in dimension = 10 and sub-manifolddimension = 100 problems.

Original: However, due to the limitation of the local sampling method, L-GSOapplies only to a single-objective function.
Revised: However, due to the limitation of the local sampling method, L-GSOapplies only to a single-objective function.

Original: Algorithm 1 GEO
Revised: Algorithm 1 GEO

Original: Require:
Revised: Require:

Original: Initial generator pool { ( G 1 , s 1 ) , ( G 2 , s 2 ) , ..., ( G p , s p ) } , initial critic networks { C 1 , C 2 , ..., C N } , a buffer memory { ( x 1 , s 1 ) , ( x 2 , s 2 ) , ..., ( x B , s B ) } , an input seed z (pseudo random constant or random).
Revised: Initial generator pool { ( G 1 , s 1 ) , ( G 2 , s 2 ) , ..., ( G p , s p ) } , initial critic networks { C 1 , C 2 , ..., C N } , a buffer memory { ( x 1 , s 1 ) , ( x 2 , s 2 ) , ..., ( x B , s B ) } , an input seed z (pseudo random constant or random).

Original: The multi-objective function is defined as ( f 1 , ..., f N ) =
Revised: The multi-objective function is defined as ( f 1 , ..., f N ) =

Original: F .
Revised: F .

Original: while iteration do while n < N do ▷ Critic network training
Revised: while iteration do while n < N do ▷ Critic network training

Original: Sample C n in { C 1 , C 2 , ..., C N } g c = ∇ θ 1 M (cid:80) Mj
Revised: Sample C n in { C 1 , C 2 , ..., C N } g c = ∇ θ 1 M (cid:80) Mj

Original: =

Revised: =

Original: || C nθ
Revised: || C nθ

Original: ( x j ) − f n ( x j ) || , x ∈ buffer
Revised: ( x j ) − f n ( x j ) || , x ∈ buffer

Original: C nθ ← Optimize
Revised: C nθ ← Optimize

Original: ( C nθ , g c ) end while while n < N do ▷ Generator network mutation (multi-objective)
Revised: ( C nθ , g c ) end while while n < N do ▷ Generator network mutation (multi-objective)

Original: Sample C n in { C 1 , C 2 , ..., C N } while m < M do ▷ M: the number of mutations
Revised: Sample C n in { C 1 , C 2 , ..., C N } while m < M do ▷ M: the number of mutations

Original: RandomSample G i in { ( G 1 , s 1 ) , ( G 2 , s 2 ) , ..., ( G p , s p ) } g g = ∇ ϕ
Revised: RandomSample G i in { ( G 1 , s 1 ) , ( G 2 , s 2 ) , ..., ( G p , s p ) } g g = ∇ ϕ

Original: [ − C n ( G i,ϕ ( z ))]
Revised: [ − C n ( G i,ϕ ( z ))]

Original: G i ← Optimize ( G i , g g ) x
Revised: G i ← Optimize ( G i , g g ) x

Original: i =
Revised: i =

Original: G i ( z ) s i
Revised: G i ( z ) s i

Original: = F ( x i ) buffer.append( ( x i , s i ) ) pool.append( ( G i , s i ) ) end while end while pool ← ParetoEfficiency(pool)
Revised: = F ( x i ) buffer.append( ( x i , s i ) ) pool.append( ( G i , s i ) ) end while end while pool ← ParetoEfficiency(pool)

Original: ▷ evolution pool ← AgeEvolution(pool)
Revised: ▷ evolution pool ← AgeEvolution(pool)

Original: ▷ (optional) end while 2.2 EGAN
Revised: ▷ (optional) end while 2.2 EGAN

Original: Evolutionary Generative Adversarial Networks (EGAN) is a method to increase Generative Adversarial Network (GAN)
Revised: Evolutionary Generative Adversarial Networks (EGAN) combines Generative Adversarial Network(GAN)

Original: [30]-[33] performance by using the evolution strategy.
Revised: [30]-[33] and evolution strategies.

Original: The core idea of EGAN isthat the evolution strategy can assume a backpropagation as a mutation.
Revised: The core idea of EGAN is that the evolution strategy canassume a backpropagation as a mutation.

Original: It has one discriminator andmulti generators.
Revised: It has one discriminator and multi generators.

Original: Generators of the evolution pool are mutated for each interation, and they are sortedby fitness scores.
Revised: Generatorsof the evolution pool are mutated for each interation, and they are sorted by fitness scores.

Original: By comparing the mode collapsing results, the study shows that the evolution strategy efficientlycomplements the GAN algorithm.
Revised: By comparing the mode collapsing results, the study shows that the evolution strategy efficientlycomplements the GAN algorithm.

Original: EGAN cannot be regarded as a black-box optimization becausethe discriminator must be trained by prepared true data (image dataset).
Revised: EGAN is not an optimizer.

Original: Nevertheless, we expectedthat a combination of evolution strategy and GAN could be adopted in our black-box optimizationalgorithm.
Revised: Nevertheless, we expected thata combination of evolution strategy and GAN could be adopted in our black-box optimizationalgorithm.

Original: 2.3 Other approaches
Revised: 2.3 Other approaches

Original: Global Topology Optimization network (GLOnet) [23] is a method fordesigning optimized electromagnetic devices.
Revised: Global Topology Optimization network (GLOnet) [23] is a method for electromagnetic device designs.

Original: It is an advanced study from the previous research, adjoint-based topologyoptimizer (ABTO)
Revised: It is an advanced study of the previous research, adjoint-based topology-optimizer (ABTO)

Original: ABTO finds the refractive index (target variables) directly from thegradient given by a device (simulator).

Original: GLOnet increases optimization performance by addinggenerator networks on ABTO.
Revised: GLOnet increases optimization performance by adding generator networks on ABTO.

Original: GLOnet is not a black-box optimizer because the gradient is givendirectly from the target simulator.
Revised: GLOnet is nota black-box optimizer because the gradient is given directly from the target simulator.

Original: However, we can discover an essential role of the generator networkfor better optimization.
Revised: However, wecan discover an essential role of the generator network for better optimization.

Original: GNN-ES (Evolutionary Strategies with Generative Neural Networks)
Revised: GNN-ES (Evolutionary Strategies with Generative Neural Networks)

Original: [28] is a combined method ofbijective neural networks and evolutions.
Revised: [28] is a combined method ofbijective neural networks and evolutions.

Original: GNN-ES assumes latent space z and bijective Generative
Revised: GNN-ES assumes latent space z and bijective Generative

Revised: Network (GNN) x

Revised: = g ( z ) , z = h ( x ) .

Revised: It optimizes latent space and bijective network, while updatinglatent space is carried out by evolution strategies.

Revised: The study shows that GNN-ES can optimize testfunctions in dimension = 10 .

Revised: GNN-ES is not a surrogate model-based optimizer and is restricted tobijective networks.

Original:  [Figure]
Revised: Conservative Objective Models (COMs) [29] is a surrogate model-based black-box optimizer. Thekey idea of COMs is regularizing the loss function of a surrogate model in training. Along with astandard supervised regression, it adds COMs-regularizer to prevent erroneously large predictions ofthe trained model. [Figure]

Original: ASB
Revised: ASB

Revised: Linear

Original: ASB
Revised: ASB

Revised: ASB

Revised: ASB

Original: Linear …
Revised: Linear …

Revised: Generator Network

Original: [Figure]
Revised: [Figure]

Revised: Branches

Original: Linear
Revised: Linear

Original: ASB ASB
Revised: ASB ASB

Original: ASB ASB
Revised: ASB ASB

Original: Linear, Activation
Revised: Linear, Activation

Original: ASB
Revised: ASB

Original: Critic Network Branches
Revised: Critic Network Branches

Original: Linear
Revised: Linear

Original: ASB ASB
Revised: ASB ASB

Original: ASB ASB
Revised: ASB ASB

Original: Linear, Activation …
Revised: Linear, Activation …

Original: Concat.
Revised: Concat.

Original: Linear, Dropout, Act.
Revised: Linear, Dropout, Act.

Original: [Figure] Network (GNN) x = g ( z ) , z = h ( x ) . It optimizes latent space and bijective network, while updatinglatent space is carried out by evolution strategies.
Revised: Linear, Dropout, Act.

Original: The study shows that GNN-ES can optimize testfunctions in dimension = 10 . GNN-ES is not a surrogate model-based optimizer and is restricted tobijective networks.
Revised: Linear

Original: Conservative Objective Models (COMs) [29] is a surrogate model-based black-box optimizationmethod. The key idea of COMs is regularizing the loss function of a surrogate model in training. Along with a standard supervised regression, it adds COMs-regularizer to prevent erroneously largepredictions of the trained model.
Revised: [Figure]

Original: Surrogate assisted evolution models is also related [58].
Revised: Surrogate assisted evolution models are also related [58].

Original: However, the algorithm does not guarantee O ( n ) computational complexity.
Revised: However, they do not guarantee O ( n )computational complexity.

Original: Because high-dimensional optimization requires a lot of functioncalls, slow speed can be the weakness.
Revised: This can be a weakness in high-dimensional problems.

Original:  Methods
Revised:  Methods

Original: [Figure]

Original: GEO consists of two stages: the evolution and network training.
Revised: GEO consists of two stages: the evolution and network training.

Original: The evolution pool maintainsa certain number of generators on the basis of the fitness score.
Revised: The evolution pool maintainsa certain number of generators on the basis of the fitness score.

Original: The evaluation and sorting ofmulti-objective scores is determined by Pareto-efficiency.
Revised: The evaluation and sorting ofmulti-objective scores is determined by Pareto-efficiency.

Original: A generator training is also a mutation in evolution.
Revised: A generator training is also a mutation in evolution.

Original: A generator is randomly sampled from the pool.2.
Revised: A generator is randomly sampled from the pool.2.

Original: The critic network trains the selected generator (using backpropagation, to increase prediction ofcritic).
Revised: The critic network trains the selected generator (using backpropagation, to increase prediction ofcritic).

Original: The trained generator suggests a new variable x = G ( z ).
Revised: The trained generator suggests a new variable x = G ( z ).

Original: Check score =
Revised: Check score =

Original: F ( x ).
Revised: F ( x ).

Original: Sort anew ( G , score ) pair in the pool.
Revised: Sort anew ( G , score ) pair in the pool.

Original: ( x , score ) pairs are stored in buffer memory and used to train the critic network.
Revised: ( x , score ) pairs are stored in buffer memory and used to train the critic network.

Original: A critic network istrained to surrogate a corresponding black-box.
Revised: A critic network istrained to surrogate a corresponding black-box.

Original: 3.1 Generative model
Revised: 3.1 Generative model

Original: GEO consists of a pool of generator networks and critic networks that create backpropagations.
Revised: GEO consists of a pool of generator networks and critic networks that create backpropagations.

Original: For each iteration step, generators are randomly sampled to make mutated generators.
Revised: For each iteration step, generators are randomly sampled to make mutated generators.

Original: N -criticnetworks are prepared to make an N -objective surrogate model, each critic network corresponds toa single objective function.
Revised: N -criticnetworks are prepared to make an N -objective surrogate model, each critic network corresponds toa single objective function.

Original: The mutated generators create variables x = G ϕ ( z ) , and N -objectivescores s i
Revised: The mutated generators create variables x = G ϕ ( z ) , and N -objectivescores s i

Original: = f i ( x ) are measured.
Revised: = f i ( x ) are measured.

Original: The set of scores ( x, s ) , s =
Revised: The set of scores ( x, s ) , s =

Original: ( s 1 , s 2 , ..., s N ) is stored in the buffermemory.
Revised: ( s 1 , s 2 , ..., s N ) is stored in the buffermemory.

Original: Training of the critic network is carried out using the buffer memory.
Revised: Training of the critic network is carried out using the buffer memory.

Original: After training, criticnetworks mutate the generator networks in the next iteration step.
Revised: After training, criticnetworks mutate the generator networks in the next iteration step.

Original: The backpropagation servesoptimal mutations by increasing predictions of critic networks.
Revised: The backpropagation servesoptimal mutations by increasing predictions of critic networks.

Original: [Equation]
Revised: [Equation]

Original: [Figure] single Usually, traditional GAN generators feed random latent variables z through the input layer, whilesome GAN algorithms separate latent variables from the input feeds [33].
Revised: Usually, traditional GAN generators feed random latent variables z through the input layer, whilesome GAN algorithms separate latent variables from the input feeds [33].

Original: Because GEO does notneed inferences, we do not see z as a latent vector.
Revised: Because GEO does notneed inferences, we do not see z as a latent vector.

Original: We experimented with both random variables(Figure 5) and pseudo-random constants (Figure 4) as input feeds z .
Revised: We experimented with both random variables(Figure 5) and pseudo-random constants (Figure 4) as input feeds z .

Original: Since each critic network has a corresponding objective, it must be trained separately using itscorresponding objective function.
Revised: Since each critic network has a corresponding objective, it must be trained separately using itscorresponding objective function.

Original: We used L1 loss with a single objective function f j and a criticnetwork C j .
Revised: We used L1 loss with a single objective function f j and a criticnetwork C j .

Original: The loss function is defined as follows
Revised: The loss function is defined as follows

Original: [Equation]
Revised: [Equation]

Original: [Equation] 
Revised: [Equation] [Figure] performances dimension

Revised: Computational time in 8,192 dimension

Revised: real

Revised: Optimization performances

Revised: and

Revised: in single-objective functions.

Revised: LSM is a modification of L-GSO.

Original: The critic network learns variable x in a global region.
Revised: The critic network learns variable x in a global region.

Original: Global training is essential for multidimensional Pareto front searches.
Revised: Global training is essential for multidimensional Pareto front searches.

Original: (See 3.4)
Revised: (See 3.4)

Original: 3.2 Evolutionstrategy
Revised: 3.2 Evolution

Revised: strategy

Original: Each generator is stored in the pool with a corresponding fitness score s j
Revised: Each generator is stored in the pool with a corresponding fitness score s j

Original: = ( s 1 j , ...s Nj )
Revised: = ( s 1 j , ...s Nj )

Original: =
Revised: =

Original: F ( x j ) .
Revised: F ( x j ) .

Original: Score data { ( x j , s j ) ... } are sorted and selected by Pareto efficiency at every iteration.
Revised: Score data { ( x j , s j ) ... } are sorted and selected by Pareto efficiency.

Original: Optionally,age evolution can be added in the sorting part.
Revised: Optionally, age evolution canbe added in the sorting part.

Original: The age evolution removes the oldest elements fromthe pool.
Revised: The age evolution removes the oldest elements from the pool.

Original: Thereby, it prevents "the high score due to stochasticity" from surviving in the pool.
Revised: Thereby,it prevents "the high score due to stochasticity" from surviving in the pool.

Original: Apool-refresh method is another option, but it make the calculation time almost doubled.
Revised: A pool-refresh method isanother option, but it makes the calculation time almost doubled.

Original: In GEO, evolution strategy is not just an auxiliary tool.
Revised: In GEO, evolution strategy is not just an auxiliary tool.

Original: Without an evolution strategy, the training ofnetworks can be unstable, which leads to the failure of optimization.
Revised: Without an evolution strategy, the training ofnetworks can be unstable, which leads to the divergence.

Original: We discuss details in section 3.4.
Revised: We discuss details in section 3.4.

Original: 3.3 Neural networks
Revised: 3.3 Neural networks

Original: All kinds of neural networks, including Recurrent Neural 
Revised: Any kinds of neural networks, including Recurrent Neural Network (RNN) [47][48][49], Convolutional Neural Network (CNN) [46], and Full Connected (FC) network can be used as generatornetworks and critic networks.

Original: [Figure]
Revised: We chose a multi-head-self-attention network [50] for operationalconvenience.

Original: Figure 2 shows the self-attention network we used.
Revised: Figure 2 shows the self-attention network we used.

Original: The overall structure is modified from the originaltransformer model.
Revised: The overall structure is modifiedfrom the original transformer model.

Original: Because the attention network consumes gigantic memory size, it may cause GPU out-of-memory.
Revised: Because the attention network consumes gigantic memory size, it may cause GPU out-of-memory.

Original: Itwas a significant problem when we optimized high-dimensional functions.
Revised: Itwas a significant problem when we optimized high-dimensional functions.

Original: When using an NVIDIA
Revised: When using an NVIDIA

Original: Tesla V100 32G GPU in variable space of dimension d > 2 11 , the total required memory exceeds theavailable memory size.
Revised: Tesla V100 32G GPU in variable space of dimension d > 2 11 , the total required memory exceeds theavailable memory size.

Original: We devised an ad hoc trunk-branch network structure to solve this problem.
Revised: We devised an ad hoc trunk-branch network structure to solve this problem.

Original: This structure has one trunk network and several branch networks, and each branch network extendsfrom the trunk network.
Revised: This structure has one trunk network and several branch networks, and each branch network extendsfrom the trunk network.

Original: Branches have an identical structure with each other, and the length of theoutput tensor is defined by
Revised: Branches have an identical structure with each other, and the length of theoutput tensor is defined byn subvar =

Original: [Equation]
Revised: [Figure]

Original: The split branch trick serves a memory-efficient structure, but it could be detrimental to optimizationperformance.
Revised: The split branch trick serves a memory-efficient structure, but it could be detrimental to optimizationperformance.

Original: Therefore, we implement the trunk-branch structure only for the sake of memoryefficiency of GPU.
Revised: Therefore, we implement the trunk-branch structure only for the sake of memoryefficiency of GPU.

Original: The baseline attention network structure includes dropout layers [51], and the dropout layers’ randomness makes the generator stochastic.
Revised: The baseline attention network structure includes dropout layers [51], and the dropout layers’ randomness makes the generator stochastic.

Original: A random input feed z is also a source of stochastic behavior.
Revised: A random input feed z is also a source of stochastic behavior.

Original: We experimented with both stochastic generators (Figure 5) and non-stochastic generators (Figure 4).
Revised: We experimented with both stochastic generators (Figure 5) and non-stochastic generators (Figure 4).

Original: However, for critic networks, we maintained the dropout layers as non-zero.
Revised: However, for critic networks, we maintained the dropout layers as non-zero.

Original: 3.4 A complementary strategy of generative network and evolution
Revised: 3.4 A complementary strategy of generative network and evolution

Original: Figure 1 shows the full algorithm of GEO.
Revised: Figure 1 shows the full algorithm of GEO.

Original: Training the surrogate model (critic network) is theessential part of a surrogate model-based optimizer, but it is also the trickiest part.
Revised: Training the surrogate model (critic network) is theessential part of a surrogate model-based optimizer, but it is also the trickiest part.

Original: The point is thatthe training data (the true data in GAN concepts) is not prepared, and it can be only acquired throughan on-the-fly search.
Revised: The point is thatthe training data (the true data in GAN concepts) is not prepared, and it can be only acquired throughan on-the-fly search.

Original: Without prepared data, training can be unstable since outbreaks of new datamake the training region fluctuate.
Revised: Without prepared data, training can be unstable since outbreaks of new datamake the training region fluctuate.

Original: In this case, the algorithm diverges for the following reasons:
Revised: In this case, the algorithm diverges for the following reasons:

Original: The generator suggests an input variable x in a wrong direction.
Revised: The generator suggests an input variable x in a wrong direction.

Original: The critic network is trained
Revised: The critic network is trained

Original: with input variable x , but x has no information of a Pareto front.
Revised: with input variable x , but x has no information of a Pareto front.

Original: The critic network trains the
Revised: The critic network trains the

Original: generator, but it does not give meaningful information.
Revised: generator, but it does not give meaningful information.

Original: In short, the divergence is a result of evil cycles of twonetworks.
Revised: In short, the divergence is a result of evil cycles of two

Original: L-GSO uses the local sampling method to stabilize a surrogate network. The local sampling methodmeans that they samples data around a center, and the center is the current point.
Revised: networks.

Original: The current point of1-objective function is a single point, it is also The Pareto-front of 1-objective function. However, it can be challenging in N -objective functions.
Revised: The local sampling method of L-GSO seems to be a simple stabilization, but, it can be challengingin N -objective functions.

Original: Since the Pareto front is R N − 1 surface, itrequires N − 1 dimensional sampling region.
Revised: Since the Pareto front is R N − 1 surface, it requires N − 1 dimensionalsampling region.

Original: There will be a lot of centers of sampling.
Revised: There will be a lot of centers of sampling.

Original: Then, it isnot local anymore.
Revised: Then, it is not local anymore.

Original: The local sampling method cannot be used in multi-objective problems.
Revised: The localsampling method cannot be used in multi-objective problems.

Original: Therefore, we need to devise a stabilizing method for multi-objective functions.
Revised: Therefore, we need to devise a stabilizing method for multi-objective functions.

Original: We suggest that theevolution strategy can be a good solution.
Revised: We suggest that theevolution strategy can be a good solution.

Original: The role of evolution is to trap G and corresponding xnear the Pareto front.
Revised: The role of evolution is to trap G and corresponding xnear the Pareto front.

Original: At the same time, the data that is far from the Pareto front is discarded.
Revised: At the same time, the data that is far from the Pareto front is discarded.

Original: Also,it slows down the fluctuation of the pool.
Revised: Also,it slows down the fluctuation of the pool.

Original: As a result, the training data region becomes stabilizedaround the Pareto front.
Revised: As a result, the training data region becomes stabilizedaround the Pareto front.

Original: We expect the stabilization method to serve as an anchor, which prevents the training data fromfloating.
Revised: We expect the stabilization method to serve as an anchor, which prevents the training data fromfloating.

Original: At the same time, a properly trained critic network provides better mutation strategies.
Revised: At the same time, a properly trained critic network provides better mutation strategies.

Original: Theinterdependent cooperation of evolution strategy and surrogate model is our main idea.
Revised: Theinterdependent cooperation of evolution strategy and surrogate model is our main idea.

Revised: [Table]

Original:  Experimental results
Revised:  Experimental results

Original: In this section, we compare the black-box optimization results of GEO with baseline optimizers.
Revised: In this section, we compare the black-box optimization results of GEO with baseline optimizers.

Original: Wetested single and multi-objective functions, stochastic and non-stochastic functions.
Revised: Wetested single and multi-objective functions, stochastic and non-stochastic functions.

Original: The baseline optimizers are Bayesian optimization (Gaussian process), NSGA-II (NSGA2) [7], NSGA-III (NSGA3)[15], MOEA/D [18], and CMA-ES evolution algorithms.
Revised: The baseline optimizers are Bayesian optimization (Gaussian process), NSGA-II (NSGA2) [7], NSGA-III (NSGA3)[15], MOEA/D [18], and CMA-ES evolution algorithms.

Original: For the implementation of NSGA-II and
Revised: For the implementation of NSGA-II and

Original: CMA-ES, we used the Pymoo package [52].
Revised: CMA-ES, we used the Pymoo package [52].

Original: 4.1 Single objective functions
Revised: 4.1 Single objective functions

Original: Figure 3a) shows performances of optimizers according to dimensions.
Revised: Figure 3a) shows performances of optimizers according to dimensions.

Original: For the single-objective testfunction, we used Styblinski-Tang function.
Revised: For the single-objective testfunction, we used Styblinski-Tang function.

Original: At dimension = 2 , Bayesian optimization shows thebest performance.
Revised: At dimension = 2 , Bayesian optimization shows thebest performance.

Original: However, as the dimension increases, GEO outperforms baseline optimizers.
Revised: However, as the dimension increases, GEO outperforms baseline optimizers.

Original: Atdimension = 8 , 192 , baseline optimizers rarely finds the global minimum, while the GEO showsbetter performances.
Revised: Atdimension = 8 , 192 , baseline optimizers rarely finds the global minimum, while the GEO showsbetter performances.

Original: To see how the depth of the generator network affects the performance, we also tested GEO with asingle-layer generator.
Revised: To see how the depth of the generator network affects the performance, we also tested GEO with asingle-layer generator.

Original: The single-layer generator appears to have little optimization capability.
Revised: The single-layer generator appears to have little optimization capability.

Original: Evenin the dimension = 2 problem, it shows a considerably slow optimization.
Revised: Evenin the dimension = 2 problem, it shows a considerably slow optimization.

Original: Bayesian optimization is a powerful method for high-cost black-box problems, but its performancecan be weakened when the problem requires numerous function calls.
Revised: Bayesian optimization is a powerful method for high-cost black-box problems, but its performancecan be weakened when the problem requires numerous function calls.

Original: The computational complexityof Bayesian optimization is known to be O ( n 3 ) [3].
Revised: The computational complexityof Bayesian optimization is known to be O ( n 3 ) [3].

Original: Figure 3b) shows the computational time of
Revised: Figure 3b) shows the computational time of

Original: GEO, NSGA-II, and Bayesian optimization.
Revised: GEO, NSGA-II, and Bayesian optimization.

Original: Like most evolution strategies, GEO is designed to havea computational complexity of O ( n ) .
Revised: Like most evolution strategies, GEO is designed to havea computational complexity of O ( n ) .

Original: Figure 3c) shows performance comparisons to a Local Surrogate Model (LSM), a modification of the
Revised: Figure 3c) shows performance comparisons to a Local Surrogate Model (LSM), a modification of the

Original: L-GSO algorithm.
Revised: L-GSO algorithm.

Original: LSM follows the general outline of L-GSO GAN implementation but adopts theself-attention network used in GEO.
Revised: LSM follows the general outline of L-GSO GAN implementation but adopts theself-attention network used in GEO.

Original: In the Styblinski-Tang function, LSM shows worse performance [Table] than in other functions.
Revised: In the Styblinski-Tang function, LSM shows worse performancethan in other functions.

Original: Also, the performance of LSM rapidly decreases as the dimension increases.
Revised: Also, the performance of LSM rapidly decreases as the dimension increases.

Original: On the other hand, GEO shows better optimization performance under various conditions.
Revised: On the other hand, GEO shows better optimization performance under various conditions.

Original: 4.2 Multi objective functions
Revised: 4.2 Multi objective functions

Original: In this section, we show optimization performance comparisons in multi-objective problems.
Revised: In this section, we show optimization performance comparisons in multi-objective problems.

Original: Weonly compare GEO and evolution strategies because the high-dimensional problems require a lot offunction calls.
Revised: Weonly compare GEO and evolution strategies because the high-dimensional problems require a lot offunction calls.

Original: Figure 4 shows optimization results in the non-stochastic-multi-objective functions.
Revised: Figure 4 shows optimization results in the non-stochastic-multi-objective functions.

Original: Each figure is a
Revised: Each figure is a

Original: Pareto front plot after 100,000 function calls.
Revised: Pareto front plot after 100,000 function calls.

Original: For the multi-objective test functions, ZDT functionsand combined functions F =
Revised: For the multi-objective test functions, ZDT functionsand combined functions F =

Original: [ f 1 , f 2 ] ( f i : Styblinski-Tang, Ackley and Rastrigin function) wereused.
Revised: [ f 1 , f 2 ] ( f i : Styblinski-Tang, Ackley and Rastrigin function) wereused.

Original: A Latin Hyper Cube (LHC) [53] method is a good guess for initial states.
Revised: A Latin Hyper Cube (LHC) [53] method is a good guess for initial states.

Original: However, GEOcannot implement LHC because it generates initial points x through the neural network.
Revised: However, GEOcannot implement LHC because it generates initial points x through the neural network.

Original: We gave twoinitial states in the baseline optimizer to control performance according to the initial state.
Revised: We gave twoinitial states in the baseline optimizer to control performance according to the initial state.

Original: NSGA 2 - Iuses GEO’s initial distribution x ∈ G ( z ) as its initial points, while NSGA 2 - LHC uses LHC-initialpoints.
Revised: NSGA 2 - Iuses GEO’s initial distribution x ∈ G ( z ) as its initial points, while NSGA 2 - LHC uses LHC-initialpoints.

Original: In most cases, GEO outperforms baseline optimizers.
Revised: In most cases, GEO outperforms baseline optimizers.

Original: The choice of initial points for NSGA-II rarelyaffects final results.
Revised: The choice of initial points for NSGA-II rarelyaffects final results.

Original: A slightly different result appears in ZDT2.
Revised: A slightly different result appears in ZDT2.

Original: In ZDT2, GEO finds an optimalpoint, but it fails to find a global shape of the Pareto front.
Revised: In ZDT2, GEO finds an optimalpoint, but it fails to find a global shape of the Pareto front.

Original: Figure 5 shows optimization results in the stochastic-multi-objective functions.
Revised: Figure 5 shows optimization results in the stochastic-multi-objective functions.

Original: The optimization ofstochastic functions is defined as follows
Revised: The optimization ofstochastic functions is defined as follows

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: [Equation]

Original: N is a random normal distribution, and we set µ = 0 . 0 and σ = 1 . 0 .
Revised: N is a random normal distribution, and we set µ = 0 . 0 and σ = 1 . 0 .

Original: Figure 5 shows optimizationresult of stochastic-multi-objective functions.
Revised: Figure 5 shows optimizationresult of stochastic-multi-objective functions.

Original: GEO outperforms baseline optimizers even in astochastic environment, but it still has a single-point collapsing problem in the ZDT2 function.
Revised: GEO outperforms baseline optimizers even in astochastic environment, but it still has a single-point collapsing problem in the ZDT2 function.

Original: 4.3 Cartpole-v
Revised: 4.3 Cartpole-v1

Original: For the variety of test functions, we optimized the OpenAI [54] Cartpole-V1 by converting it into ablack-box problem.
Revised: For the variety of test functions, we optimized the OpenAI [54] Cartpole-V1 by converting it into ablack-box problem.

Original: It is also a simple toy model of time-sequential input-output (I/O) problems.
Revised: It is also a simple toy model of time-sequential input-output (I/O) problems.

Original: Cartpole-V1 is a test package that is mainly used in reinforcement learning [34][35].
Revised: Cartpole-V1 is a test package that is mainly used in reinforcement learning [34][35].

Original: Reinforcementlearning requires a series of I/O structure.
Revised: Reinforcementlearning requires a series of I/O structure.

Original: However, in the black-box problem, the entire inputsequence is assumed as one large input, and only the final score is measured without observing theintermediate rewards and states.
Revised: However, in the black-box problem, the entire inputsequence is assumed as one large input, and only the final score is measured without observing theintermediate rewards and states.

Original: Since this experiment is a toy model of real-world problems whichhave stochastic environments, we kept the Cartpole-V1 stochastic.
Revised: Since this experiment is a toy model of real-world problems whichhave stochastic environments, we kept the Cartpole-V1 stochastic.

Original: The final score is measured in a relative score to the sequence length.
Revised: The final score is measured in a relative score to the sequence length.

Original: (If the Cartpole is alive form -length in n -sequences, the score is m/n .)
Revised: (If the Cartpole is alive form -length in n -sequences, the score is m/n .)

Original: Therefore, the maximum and minimum score set to 1 .
Revised: Therefore, the maximum and minimum score set to 1 .

Original: 0and 0 . 0 .
Revised: 0and 0 . 0 .

Original: GEO outperforms others from 256 to 2048 dimensions in the experiment (Table 1).
Revised: GEO outperforms others from 256 to 2048 dimensions in the experiment (Table 1).

Original: Cartpole-V1 is difficult to solve with black-box optimization approach due to the stochastic changein the initial state, but it shows a clear performance difference between the black-box optimizers,nonetheless.
Revised: Cartpole-V1 is difficult to solve with black-box optimization approach due to the stochastic changein the initial state, but it shows a clear performance difference between the black-box optimizers,nonetheless.

Original: [s = 0.999] 1 : [s = 0.959] 2 : [s = 0.999] 3 : [s = 0.999] 4 : [s = 0.999]
Revised: 4.4 LeNet-5

Original: We also optimized LeNet-5, which is trained with the MNIST dataset.
Revised: We also optimized LeNet-5, which is trained with the MNIST dataset.

Original: The optimization goal is togenerate an image that makes the LeNet-5 predict a target number with a maximum score (maximumprediction score = 1 . 0 ).
Revised: The optimization goal is togenerate an image that makes the LeNet-5 predict a target number with a maximum score (maximumprediction score = 1 . 0 ).

Original: The LeNet-5 is regarded as a non-differentiable black-box.
Revised: The LeNet-5 is regarded as a non-differentiable black-box.

Original: After 50 , 000function calls, the final scores of generated images reach very close to the maximum score (Figure 6).
Revised: After 50 , 000function calls, the final scores of generated images reach very close to the maximum score (Figure 6).

Original: In the related experiment, L-GSO, generative models appear to be better at finding the local optimumin sub-manifolds.
Revised: In the related experiment, L-GSO, generative models appear to be better at finding the local optimumin sub-manifolds.

Original: For the same reason, we expected to see the sub-manifold structure in the generatedimage, but the generated image does not seem intuitive.
Revised: For the same reason, we expected to see the sub-manifold structure in the generatedimage, but the generated image does not seem intuitive.

Original:  Discussion
Revised:  Discussion

Original: Often, a neural network’s learning mechanism is likened to learning manifolds in a high-dimensionalspace.
Revised: Often, a neural network’s learning mechanism is likened to learning manifolds in a high-dimensionalspace.

Original: Similarly, we guess that the critic network in GEO learns low-dimensional manifolds in thehigh-dimensional space.
Revised: Similarly, we guess that the critic network in GEO learns low-dimensional manifolds in thehigh-dimensional space.

Original: Therefore, we expect that finding global or local optima would be easy ifthe optima are in low-dimensional manifolds.
Revised: Therefore, we expect that finding global or local optima would be easy ifthe optima are in low-dimensional manifolds.

Original: Also, we consider that it is the reason why GEO withsingle-layer generators optimizes a target very slowly.
Revised: Also, we consider that it is the reason why GEO withsingle-layer generators optimizes a target very slowly.

Revised: [s = 0.999] 1 : [s = 0.959] 2 : [s = 0.999] 3 :

Revised: [s = 0.999] 4 : [s = 0.999]

Original: Stabilization of critic network training is another key.
Revised: Stabilization of critic network training is another key.

Original: Although we implemented the evolutionstrategy in this study, any other methods which can stabilize the training process would be helpful.
Revised: Although we implemented the evolutionstrategy in this study, any other methods which can stabilize the training process would be helpful.

Original:  Conclusion
Revised:  Conclusion

Original: We have described a method for stochastic-multi-objective black-box optimization.
Revised: We have described a method for stochastic-multi-objective black-box optimization.

Original: GEO is aninterdependent cooperation method of generative neural networks and the evolution strategy.
Revised: GEO is aninterdependent cooperation method of generative neural networks and the evolution strategy.

Original: Theevolution strategy provides a stable training region for critic networks, and the critic networks provideefficient mutations to the evolution strategy.
Revised: Theevolution strategy provides a stable training region for critic networks, and the critic networks provideefficient mutations to the evolution strategy.

Original: As our design intent, GEO seems to work appropriatelyin stochastic and high-dimensional multi-objective test functions.
Revised: As our design intent, GEO seems to work appropriatelyin stochastic and high-dimensional multi-objective test functions.

Original: Meanwhile, the Pareto front collapsing problem, shown in ZDT2, is an important issue to be dealtwith.
Revised: Meanwhile, the Pareto front collapsing problem, shown in ZDT2, is an important issue to be dealtwith.

Original: Another limitation of GEO is the GPU memory consumption problem.
Revised: Another limitation of GEO is the GPU memory consumption problem.

Original: The excessive memoryconsumption of attention networks limits its search space to around 10 , 000 dimensions.
Revised: The excessive memoryconsumption of attention networks limits its search space to around 10 , 000 dimensions.

Original: In futureresearch, we can study other memory-efficient networks to solve this problem.
Revised: In futureresearch, we can study other memory-efficient networks to solve this problem.

Original: GEO is designed for optimization in a very high dimension.
Revised: GEO is designed for optimization in a very high dimension.

Original: However, the performance at lowerdimensions is not guaranteed (see supplement).
Revised: However, the performance at lowerdimensions is not guaranteed (see supplement).

Original: We think that d = [1,000 to 10,000] can be thepractical range of use of GEO.
Revised: We think that d = [1,000 to 10,000] can be thepractical range of use of GEO.

Original: In addition, GEO is designed to find local-optima, not a globaloptimum.
Revised: In addition, GEO is designed to find local-optima, not a globaloptimum.

Original: It is because the mutation of generators only concetrates on an exploit.
Revised: It is because the mutation of generators only concetrates on an exploit.

Original: In the next study, astrong exploit & explore strategy (like what BO has) should be added to make better optimizationcapability.
Revised: In the next study, astrong exploit & explore strategy (like what BO has) should be added to make better optimizationcapability.

Original: Checklist
Revised: Checklist

Original: For all authors...
Revised: For all authors...

Original: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’scontributions and scope?
Revised: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’scontributions and scope?

Original: [Yes](b) Did you describe the limitations of your work?
Revised: [Yes] (b) Did you describe the limitations of your work?

Original: [Yes] See discussion and conclusion(c)
Revised: [Yes] See discussion and conclusion

Original:  Did you discuss any potential negative societal impacts of your work?
Revised: (c) Did you discuss any potential negative societal impacts of your work?

Original: [Yes](d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?
Revised: [Yes] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?

Original: [Yes]
Revised: [Yes]

Original: If you are including theoretical results...
Revised: If you are including theoretical results...

Original: (a) Did you state the full set of assumptions of all theoretical results?
Revised: (a) Did you state the full set of assumptions of all theoretical results?

Original: [N/A]
Revised: [N/A]

Original: (b) Did you include complete proofs of all theoretical results?
Revised: (b) Did you include complete proofs of all theoretical results?

Original: [N/A]
Revised: [N/A]

Original: If you ran experiments...
Revised: If you ran experiments...

Original: (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
Revised: (a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)?

Original: [Yes] See supplementary for implementation details
Revised: [Yes] See supplemen- tary for implementation details

Original: (b) Did you specify all the training details (e g., data splits, hyperparameters, how theywere chosen)?
Revised: (b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?

Original: [Yes] See supplementary for implementation details
Revised: [Yes] See supplementary for implementation details

Original: (c) Did you report error bars (e g., with respect to the random seed after running experi- ments multiple times)?
Revised: (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?

Original: [Yes] See supplementary for experimental details
Revised: [Yes] See supplementary for experimental details

Original: (d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
Revised: (d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?

Original: [Yes] See supplementary for experimental details
Revised: [Yes] See supplementary for experimentaldetails

Original: If you are using existing assets (e g., code, data, models) or curating/releasing new assets...
Revised: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...

Original: (a) If your work uses existing assets, did you cite the creators?
Revised: (a) If your work uses existing assets, did you cite the creators?

Original: [Yes] See references
Revised: [Yes] See references(b)

Original: (b) Did you mention the license of the assets?
Revised: Did you mention the license of the assets?

Original: [Yes] See supplementary
Revised: [Yes] See supplementary

Original: (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [No] (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
Revised: [No](d) Did you discuss whether and how consent was obtained from people whose data you’reusing/curating?

Original: [Yes]
Revised: [Yes]

Original: (e) Did you discuss whether the data you are using/curating contains personally identifiableinformation or offensive content?
Revised: (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?

Original: [Yes]
Revised: [Yes]

Original: If you used crowdsourcing or conducted research with human subjects...
Revised: If you used crowdsourcing or conducted research with human subjects...

Original: (a) Did you include the full text of instructions given to participants and screenshots, if
Revised: (a) Did you include the full text of instructions given to participants and screenshots, if

Original: applicable?
Revised: applicable?

Original: [N/A]
Revised: [N/A]

Original: (b) Did you describe any potential participant risks, with links to Institutional Review
Revised: (b) Did you describe any potential participant risks, with links to Institutional Review

Original: Board (IRB
Revised: Board (IRB) approvals, if applicable? [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?

Revised: [N/A]