Original: Generative Evolutionary Strategy For Black-Box Optimizations
Revised: Generative Evolutionary Strategy For Black-Box Optimizations

Revised: Anonymous Author(s)

Revised: AffiliationAddress

Original: Abstract
Revised: Abstract

Original: Many scientific and technological problems are related to optimization.
Revised: Many scientific and technological problems are related to optimization.

Original: Amongthem, black-box optimization in high-dimensional space is particularly challenging.
Revised: Amongthem, black-box optimization in high-dimensional space is particularly challenging.

Original: Recent neural network-based black-box optimization studies have shown noteworthy achievements.
Revised: Recent neural network-based black-box optimization studies have shown noteworthy achievements.

Original: However, their capability in high-dimensional search space isstill limited.
Revised: However, their capability in high-dimensional search spaceis still limited.

Original: This study proposes a black-box optimization method based on evolution strategy and generative neural network model.
Revised: This study proposes a black-box optimization method based onevolution strategy and generative neural network model.

Original: We designed the algorithm sothat the evolutionary strategy and the generative neural network model work cooperatively with each other.
Revised: We designed the algorithm so that the evolutionary strategy and the generative neural network modelwork cooperatively with each other.

Original: This hybrid model enables reliable training of surrogatenetworks; it optimizes multi-objective, high-dimensional, and stochastic blackbox functions.
Revised: This hybrid model enables reliable training ofsurrogate networks; it optimizes multi-objective, high-dimensional, and stochastic black-box functions.

Original: In this experiment, our method outperforms baseline optimizationmethods, including evolution strategies, anda Bayesian optimization
Revised: In this experiment, our method outperforms baselineoptimization methods, including CMAES, NSGA-II, and Bayesian optimization

Original:  Introduction
Revised:  Introduction

Original: Optimization is one of the most crucial issues in science and technology.
Revised: Optimization is one of the most crucial issues in science and technology.

Original: Various simulations andexperiments work as black-box functions, and there have also been innumerable optimization studies.
Revised: Various simulations andexperiments work as black-box functions, and there have also been innumerable optimization studies.

Original: Gradient-based optimization methods are easy choices for differentiable functions or simple convexfunctions.
Revised: Gradient-based optimization methods are easy choices for differentiable functions or simple convexfunctions.

Original: However, black-box functions are often non-differentiable and non-convex.
Revised: However, black-box functions are often non-differentiable and non-convex.

Original: Furthermore,they can be multi-objective and stochastic.
Revised: Furthermore,they can be multi-objective and stochastic.

Original: A simple description of multi-objective black-boxoptimization is as follows
Revised: A simple description of multi-objective black-boxoptimization is as follows

Original: Optimize ( f 1 ( X ) , . . . , f m ( X ))

Original: [Equation]
Revised: [Equation]

Original: The search space is defined in real space R , where N is the dimension.
Revised: The search space is defined in real space R , where N is the dimension.

Original: f i is a single-objectivefunction, which can be stochastic.
Revised: f i is a single-objectivefunction, which can be stochastic.

Original: The evaluation of electronic device designs is a practical problem of theblack-box optimization.
Revised: The evaluation of electronic device design is a practical problem of the optimization.

Original: Since many device simulators have time-sequential input-output structures, it seems like they canbe solved in reinforcement learning.
Revised: Since thedevice simulators have time-sequential input-output structures, it seems like they can be solved inreinforcement learning.

Original: However, if theobservation cost  is too high, it will be almostimpossible to observe time-sequential data.
Revised: However, if the cost of simulator is too high, it is almost impossible to observetime-sequential data.

Original: Instead, the only information which we can observe isthe final score.

Original: Therefore, the evaluation problem is defined as a black-box optimization, in this case.
Revised: Therefore, the evaluation problem is defined as a black-box optimization, inthis case.

Original: For practical purposes, researchers have studied optimization methods in various ways.
Revised: For practical purposes, researchers have studied optimization methods in various ways.

Original: Typically, Bayesian optimization [1][2][3] and evolutionary strategies [7]-[20] are widely used.
Revised: Typically, Bayesian optimization [1][2][3] and evolutionary strategies [7]-[20] are widely used.

Original: Notably, the Bayesian optimization is advantageous when the cost of the target function is high andthe number of function calls is limited.
Revised: Notably, the Bayesian optimization is advantageous when the cost of the target function is high and the number of function calls is limited.

Original: The estimating process of Bayesian optimization is very efficient when the number of search points issmall.
Revised: The estimating process of Bayesian optimization is very efficient when the number of search points is small.

Original: On the contrary, it becomes inefficient when the number of search points increases.
Revised: On the contrary, it becomes inefficient when the number of search points increases.

Original: Therefore,
Revised: Therefore,

Original: [Figure] [Figure]
Revised: Evolution Strategy Generative N.N. model Stable training

Original: [Figure]
Revised: [Figure]

Original: Bayesian optimization may not be appropriate in high-dimensional problems that require a largenumber of function calls.
Revised: Bayesian optimization may not be appropriate in high-dimensional problems that require a largenumber of function calls.

Original: Meanwhile, evolution strategies(ES) can be better choices for aforementioned cases.
Revised: Meanwhile, evolution strategies can be better choices for aforementioned cases.

Original: In most evolution strategies, the computational cost does not drastically increase according to the number ofsearch points.
Revised: In most evolutionstrategies, the computational cost does not drastically increase according to the number of searchpoints.

Original: Nevertheless, it does not mean that they can avoid the curse of dimensionality .
Revised: Nevertheless, it does not mean that they can avoid the curse of dimensionality .

Original: Theoptimization performance of the evolutionary strategies also decreases rapidly as the dimension of theblack-box increases.
Revised: The optimization performance of the evolutionary strategies also decreases rapidly as the dimension of theblack-box increases.

Original: Although SEP-CMA[55], VD-CMA[56], and LM-CMA[57] have shown optimization capability in high-dimensional space of convex functions, optimization of high-dimensionalnon-convex problems still seems difficult.
Revised: Although SEP-CMA[55], VD-CMA[56], and LM-CMA[57] have shown optimization capability in high-dimensional space of convex functions, optimization of high-dimensionalnon-convex problems still seems difficult.

Original: Generative neural network-based models are recent approaches.
Revised: Generative neural network-based models are recent approaches.

Original: They show noteworthy performancein test function optimizations, but their capability seems to be limited to single-objective functionsand to 100-dimension [21][28].
Revised: They show noteworthy performancein test function optimizations, but their capability seems to be limited to single-objective functionsand to 100-dimension [21][28].

Original: We present GEO: Generative Evolutionary Optimization, a methodfor general black-box optimizations.
Revised: We present GEO: Generative Evolutionary Optimization, a methodfor general black-box optimizations.

Original: It is designed to optimize stochastic, multi-objective, and highdimensional black-box problems.
Revised: It is designed to optimize stochastic, multi-objective, andhigh-dimensional black-box problems.

Original: We show that GEO outperforms baseline methods in finding Paretofronts of Styblinski-Tang [42], Ackley [39], Rastrigin [36][37][38], Rosenbrock [40][41],ZDT1,
Revised: We show that GEO outperforms baseline methods in finding Pareto fronts of Styblinski-Tang [42], Ackley [39], Rastrigin [36][37][38], Rosenbrock [40][41],

Original:  ZDT2, and ZDT3 [43] test functions.
Revised: ZDT1, ZDT2, and ZDT3 [43] test functions.

Original: Also, by converting Cartpole-V1 [44]  to high-dimensionalblack-box problems, we show that GEO can be used in sequential problems. We also tested LeNet-5[45] to see how it generates sub-manifold structures.
Revised: Also, by converting Cartpole-V1 [44] (and LeNet-5[45]) to high-dimensional black-box problems, we show that GEO can generate optimized sequences(and images) without observing intermediate states (and without the direct gradient flow from targetfunctions).

Original:  Related works
Revised:  Related works

Original: GEO is related to Evolutionary Generative Adversarial Networks (EGAN) [22], and Local Generative
Revised: GEO is related to Evolutionary Generative Adversarial Networks (EGAN) [22]. Like Local Generative

Original: Surrogates Optimization (L-GSO) [21].
Revised: Surrogates Optimization (L-GSO) [21], surrogate model-based optimizations are also related to GEO.

Original: This section briefly introduces  them.
Revised: This section briefly introduces the network-based black-box optimizations and EGAN to show how GEO can adapt them.

Original: 2.1 L-GSO
Revised: 2.1 L-GSO

Original: L-GSO is a surrogate network model based black-box optimizer. It has a  surrogate network and agenerator network.
Revised: The black-box optimization with gradient descent method is a tricky task since the black-box doesnot provide gradient information directly. The L-GSO study shows that a local surrogate modelcan be a good solution for it.

Original: The main idea of L-GSO is that the surrogate networkonly estimates a localshape of  the objective function.
Revised: In this study, the surrogate network estimates a local shape of theobjective function, and the locality is a crucial idea of L-GSO.

Original: Since the stabilization of the surrogate network is difficult, theysuggest only to surrogate  a local region. Also, the optimizer can be used in  stochastic environmentssince it works in the neural network.
Revised: By using local sampling of ε -cube (orball) around a currently-optimized point, a stable surrogate model provides local gradients. Anotheradvantage of L-GSO is that the neural network-based surrogate model naturally works in stochastic environments. Because surrogate networks estimate a distribution probabilistically, the model cansuggest gradients even in a stochastic function.

Original: It is shown that L-GSO outperforms baseline optimizers in dimension = 10 and sub-manifolddimension = 100 problems.
Revised: It is shown that L-GSO outperforms baselineoptimizers (Bayesian optimization and CMA-ES [9][10] in dimension = 10 and sub-manifolddimension = 100 problems).

Original: However, due to the limitation of the local sampling method, L-GSOapplies only to a single-objective function.
Revised: However, due to the limitation of the local sampling method, L-GSOapplies only to a single-objective function.

Original: Algorithm 1 GEO
Revised: Algorithm 1 GEO

Original: Require:
Revised: Require:

Original: Initial generator pool { ( G 1 , s 1 ) , ( G 2 , s 2 ) , ..., ( G p , s p ) } , initial critic networks { C 1 , C 2 , ..., C N } , a buffer memory { ( x 1 , s 1 ) , ( x 2 , s 2 ) , ..., ( x B , s B ) } , an input seed z (constant or random variable).
Revised: Initial generator pool { ( G 1 , s 1 ) , ( G 2 , s 2 ) , ..., ( G p , s p ) } , initial critic networks { C 1 , C 2 , ..., C N } , a buffer memory { ( x 1 , s 1 ) , ( x 2 , s 2 ) , ..., ( x B , s B ) } , an input seed z (pseudo random constant or random).

Original: The multi-objective function is defined as ( f 1 , ..., f N ) =
Revised: The multi-objective function is defined as ( f 1 , ..., f N ) =

Original: F .
Revised: F .

Original: while iteration do while n < N do ▷ Critic network training
Revised: while iteration do while n < N do ▷ Critic network training

Original: Sample C n in { C 1 , C 2 , ..., C N } g c = ∇ θ 1 M1 (cid:80) M 1 j =

Revised: Sample C n in { C 1 , C 2 , ..., C N } g c = ∇ θ 1 M (cid:80) Mj =

Original: || C nθ
Revised: || C nθ

Original: ( x j ) − f n ( x j ) || , x ∈ buffer
Revised: ( x j ) − f n ( x j ) || , x ∈ buffer

Original: C nθ ← Optimize
Revised: C nθ ← Optimize

Original: ( C nθ , g c ) end while while n < N do ▷ Generator network mutation (multi-objective)
Revised: ( C nθ , g c ) end while while n < N do ▷ Generator network mutation (multi-objective)

Original: Sample C n in { C 1 , C 2 , ..., C N } while m < M2 do▷ M 2 : the number of mutations
Revised: Sample C n in { C 1 , C 2 , ..., C N } while m < M do

Original: RandomSample G i in { ( G 1 , s 1 ) , ( G 2 , s 2 ) , ..., ( G p , s p ) } g g = ∇ ϕ
Revised: RandomSample G i in { ( G 1 , s 1 ) , ( G 2 , s 2 ) , ..., ( G p , s p ) } g g = ∇ ϕ

Original: [ ± C n ( G i,ϕ ( z ))]
Revised: [ − C n ( G i,ϕ ( z ))]

Original: ▷ +: minimize, -:maximize

Original: G i ← Optimize ( G i , g g ) x
Revised: G i ← Optimize ( G i , g g ) x

Original: i =
Revised: i =

Original: G i ( z ) s i
Revised: G i ( z ) s i

Original: = F ( x i ) buffer.append( ( x i , s i ) ) pool.append( ( G i , s i ) ) end while end while pool ← ParetoEfficiency(pool) ▷ Non-dominated sorting pool ← AgeEvolution(pool)
Revised: = F ( x i ) buffer.append( ( x i , s i ) ) pool.append( ( G i , s i ) ) end while end while pool ← ParetoEfficiency(pool) ▷ evolution pool ← AgeEvolution(pool)

Original: ▷ (optional) end while
Revised: ▷ (optional) end while

Original: 2.2 EGAN
Revised: 2.2 EGAN

Original: Evolutionary Generative Adversarial Networks (EGAN) combines Generative Adversarial Network(GAN)
Revised: Evolutionary Generative Adversarial Networks (EGAN) is a method to increase Generative Adversarial Network (GAN)

Original: [30]-[33] and evolution strategies.
Revised: [30]-[33] performance by using the evolution strategy.

Original: The core idea of EGAN is that the evolution strategy canassume a backpropagation as a mutation.
Revised: The core idea of EGANis that the evolution strategy can assume a backpropagation as a mutation.

Original: It has one discriminator and multi generators.
Revised: It has one discriminatorand multi generators.

Original: Generatorsof the evolution pool are mutated  for each iteration, and they are sorted by fitness scores.
Revised: EGAN creates mutated generators for each iteration, and it sorts generators byfitness scores.

Original: By comparing the mode collapsing results, the study shows that the evolution strategy efficientlycomplements the GAN algorithm.
Revised: By comparing the mode collapsing results, the study shows that the evolution strategyefficiently complements the GAN algorithm.

Original: EGAN is not an optimizer.
Revised: EGAN cannot be regarded as a black-box optimizationbecause the discriminator must be trained by prepared true data (image dataset).

Original: Nevertheless, we expected that acombination ofan evolution strategy anda GAN could be adopted in our black-box optimizationalgorithm.
Revised: Nevertheless, weexpected that a combination of evolution strategy and GAN could be adopted in our black-boxoptimization algorithm.

Original: 2.3 Other approaches
Revised: 2.3 Other approaches

Original: Global Topology Optimization network (GLOnet) [23] is a method for  electromagnetic device designs.
Revised: Global Topology Optimization network (GLOnet) [23] is a method for designing optimized electromagnetic devices.

Original: It is an advanced study of the previous research, adjoint-based topology-optimizer (ABTO)
Revised: It is an advanced study from the previous research, adjoint-based topologyoptimizer (ABTO)

Revised: ABTO finds the refractive index (target variables) directly from thegradient given by a device (simulator).

Original: GLOnet increases optimization performance by adding generator networks on ABTO.
Revised: GLOnet increases optimization performance by addinggenerator networks on ABTO.

Original: GLOnet is nota black-box optimizer because the gradient is given directly from the target simulator.
Revised: GLOnet is not a black-box optimizer because the gradient is givendirectly from the target simulator.

Original: However, wecan discover an essential role of the generator network for better optimization.
Revised: However, we can discover an essential role of the generator networkfor better optimization.

Original: GNN-ES (Evolutionary Strategies with Generative Neural Networks)
Revised: GNN-ES (Evolutionary Strategies with Generative Neural Networks)

Original: [28] is a combined method ofbijective neural networks and evolutions.
Revised: [28] is a combined method ofbijective neural networks and evolutions.

Original: GNN-ES assumes latent space z and bijective Generative
Revised: GNN-ES assumes latent space z and bijective Generative

Original: Network (GNN) x
Revised: Network (GNN) x

Original: = g ( z ) , z = h ( x ) .
Revised: = g ( z ) , z = h ( x ) .

Original: It optimizes latent space anda bijective network. The updateof latent space is carried out by evolution strategies.
Revised: It optimizes latent space and bijective network, while updatinglatent space is carried out by evolution strategies.

Original: The study shows that GNN-ES can optimize testfunctions in dimension = 10 .
Revised: The study shows that GNN-ES can optimize test [Figure] ASB ASB

Revised: Linear …

Original: However, GNN-ES is not a surrogate model-based optimizer and it isrestricted to bijective networks.
Revised: [Figure] Linear ASB ASB

Revised: ASB

Revised: ASB

Revised: Linear, Activation

Original: Conservative Objective Models (COMs) [29] is a surrogate model-based black-box optimizer.
Revised: ASB Critic Network Branches Linear ASB

Original: Thekey idea of COMs is regularizing the loss function of a surrogate model in training.
Revised: ASB ASB ASB Linear, Activation …

Original: Along with astandard supervised regression, it adds COMs-regularizers to prevent erroneously large predictions ofthe trained model.
Revised: Concat. Linear, Dropout, Act.

Original: [Figure]
Revised: [Figure]

Original: Surrogate assisted evolution models are also related [58]. However, they do not guarantee O ( n )computational complexity.
Revised: functions in Conservative Objective Models (COMs) [29] is a surrogate model-based black-box optimizationmethod.

Original: This can be a weakness in high-dimensional problems.
Revised: The key idea of COMs is regularizing the loss function of a surrogate model in training. Along with a standard supervised regression, it adds COMs-regularizer to prevent erroneously largepredictions of the trained model.

Original:  Methods
Revised:  Methods

Original: GEO consists of two stages: the evolution and network training.
Revised: GEO consists of two stages: the evolution and network training.

Original: The evolution pool maintains acertain number of generators on the basis of  fitness scores.
Revised: The evolution pool maintainsa certain number of generators on the basis of the fitness score.

Original: The evaluation and sorting of multiobjective scores is determined by Pareto-efficiency.
Revised: The evaluation and sorting ofmulti-objective scores is determined by Pareto-efficiency.

Original: A generator training is also a mutation inthe evolution.
Revised: A generator training is also a mutation in evolution.

Original: A generator is randomly sampled fromthe pool.
Revised: A generator is randomly sampled from the pool.2.

Original: The critic network trains the selected generator (using backpropagation, to increaseordecrease a prediction of a critic).
Revised: The critic network trains the selected generator (using backpropagation, to increase prediction ofcritic).

Original: The trained generator suggests a new variable x = G ( z ).
Revised: The trained generator suggests a new variable x = G ( z ).

Original: Checkscore = F ( x ).
Revised: Check score = F ( x ).

Original: Sort a new ( G , score ) pair in the pool.
Revised: Sort anew ( G , score ) set in a pool

Original: ( x , score ) pairs are stored ina buffer memory andthey are used to train  critic networks.
Revised: ( x , score ) pairs are stored in buffer memory and used to train the critic network.

Original: Each criticnetwork is trained to surrogate a corresponding black-box object.
Revised: A critic network istrained to surrogate a black-box.

Original: 3.1 Generative model
Revised: 3.1 Generative model

Revised: [Figure]

Original: GEO consists of a pool of generator networks and critic networks that create backpropagations.
Revised: GEO consists of a pool of generator networks and critic networks that create backpropagations.

Original: For each iteration step, generators are randomly sampled to make mutated generators.
Revised: For each iteration step, generators are randomly sampled to make mutated generators.

Original: N -criticnetworks are prepared to make an N -objective surrogate model, each critic network corresponds toa single objective function.
Revised: N -criticnetworks are prepared to make an N -objective surrogate model, each critic network corresponds toa single objective function.

Original: The mutated generators create variables x = G ϕ ( z ) , and N -objectivescores s i
Revised: The mutated generators create variables x = G ϕ ( z ) , and N -objectivescores s i

Original: = f i ( x ) are measured.
Revised: = f i ( x ) are measured.

Original: The set of scores ( x, s ) , s =
Revised: The set of scores ( x, s ) , s =

Original: ( s 1 , s 2 , ..., s N ) is stored in the buffermemory.
Revised: ( s 1 , s 2 , ..., s N ) is stored in the buffermemory.

Original: Training of the critic network is carried out using the buffer memory.
Revised: Training of the critic network is carried out using the buffer memory.

Original: After training, criticnetworks mutate the generator networks in the next iteration step.
Revised: After training, criticnetworks mutate the generator networks in the next iteration step.

Original: The backpropagation servesoptimal mutations by increasing predictions of critic networks.
Revised: The backpropagation servesoptimal mutations by increasing predictions of critic networks.

Original: [Equation]
Revised: [Equation]

Original: Usually, traditional GAN generators feed random latent variables z through the input layer, whilesome GAN algorithms separate latent variables from the input feeds [33].
Revised: Usually, traditional GAN generators feed random latent variables z through the input layer, whilesome GAN algorithms separate latent variables from the input feeds [33].

Original: Because GEO does notneed inferences, we do not see z as a latent vector.

Original: We experimented with both random variables(Figure 5) and  constants (Figure 4) as input feeds z .
Revised: We experimented withboth random variables (Figure 5) and pseudo-random constants (Figure 4) as input feeds z .

Original: Since each critic network has a corresponding objective, it must be trained separately using itscorresponding objective function.
Revised: Since each critic network has a corresponding objective, it must be trained separately using itscorresponding objective function.

Original: We used L1 loss with a single objective function f j and a criticnetwork C j .
Revised: We used L1 loss with a single objective function f j and a criticnetwork C j .

Original: The loss function is defined as follows
Revised: The loss function is defined as follows

Revised: [Figure]

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: The critic network learns variable x in a global region.
Revised: The critic network learns variable x in a global region.

Original: Global training is essential for multidimensional Pareto front searches.
Revised: Global training is essential for multidimensional Pareto front searches.

Original: (See 3.4)
Revised: (See 3.4)

Original: [Figure]

Original: 3.2 Evolution strategy
Revised: 3.2 Evolution strategy

Original: [Figure]

Original: Each generator is stored in the pool with a corresponding fitness score s j
Revised: Each generator is stored in the pool with a corresponding fitness score s j

Original: = ( s 1 j , ...s Nj )
Revised: = ( s 1 j , ...s Nj )

Original: =
Revised: =

Original: F ( x j ) .
Revised: F ( x j ) .

Original: Score data { ( x j , s j ) ... } are sorted  by Paretoefficiency. The Pareto efficiency is defined as follows
Revised: Score data { ( x j , s j ) ... } are sorted and selected by Pareto efficiency at every iteration.

Original: [Equation]

Original: then, x ∗ ∈ P , where x ∈ X and P is the Pareto efficiency.

Original: Pareto efficiency can be ranked in order

Original: P 1 =

Original: Pareto( X ), P 2 =

Original: Pareto( X − P 1 ), · · · , they are calculated by non-dominated sorting methods.

Original: Amore detailed explanation is provided in the supplement.

Original: Optionally, age evolution can be added in the sorting part.
Revised: Optionally,age evolution can be added in the sorting part.

Original: The age evolution removes the oldestelements from the pool.
Revised: The age evolution removes the oldest elements fromthe pool.

Original: Thereby, it prevents "the high score due to stochasticity" from surviving inthe pool.
Revised: Thereby, it prevents accidentally-high-scoring elements from surviving in the pool.

Original: A pool-refresh method is another option, but it makes the calculation time almost doubled.

Original: In GEO, evolution strategy is not just an auxiliary tool.
Revised: In GEO, evolution strategy is not just an auxiliary tool.

Original: Without an evolution strategy, the training ofnetworks can be unstable, which leads to the divergence.
Revised: Without an evolution strategy, the training ofnetworks can be unstable, which leads to the failure of optimization.

Original: We discuss details in section 3.4.
Revised: We discuss details in section 3.4.

Original: 3.3 Neural networks
Revised: 3.3 Neural networks

Original: Any kinds of neural networks, including Recurrent Neural Network (RNN) [47][48][49], Convolutional Neural Network (CNN) [46], and Full Connected (FC) network can be used as generatornetworks and critic networks.
Revised: All kinds of neural networks, including Recurrent Neural Network (RNN) [47][48][49], Convolutional Neural Network (CNN) [46], and Full Connected (FC) network can be used as generator networksand critic networks.

Original: We chose a multi-head-self-attention network [50] for operationalconvenience.
Revised: We chose a multi-head-self-attention network [50] for operational convenience.

Original: Figure 2 shows the self-attention network we used.
Revised: Figure 2 shows the self-attention network we used.

Original: The overall structure is modifiedfrom the original transformer model.
Revised: The overall structure is modified from the originaltransformer model.

Original: Because the attention network consumes gigantic memory size, it may cause GPU out-of-memory.
Revised: Because the attention network consumes gigantic memory size, it may cause GPU out-of-memory.

Original: Itwas a significant problem when we optimized high-dimensional functions.
Revised: Itwas a significant problem when we optimized high-dimensional functions.

Original: When using an NVIDIA
Revised: When using an NVIDIA

Original: Tesla V100 32G GPU in variable space of dimension d > 2 11 , the total required memory exceeds theavailable memory size.
Revised: Tesla V100 32G GPU in variable space of dimension d > 2 11 , the total required memory exceeds theavailable memory size.

Original: We devised an ad hoc trunk-branch network structure to solve this problem.
Revised: We devised an ad hoc trunk-branch network structure to solve this problem.

Original: [Figure]
Revised: [Figure]

Original: This structure has one trunk network and several branch networks, and each branch network extendsfrom the trunk network.
Revised: This structure has one trunk network and several branch networks, and each branch network extendsfrom the trunk network.

Original: Branches have an identical structure with each other, and the length of theoutput tensor is defined by n subvar = n var /n branches .
Revised: Branches have an identical structure with each other, and the length of theoutput tensor is defined by [Equation]

Original: The split branch trick serves a memory-efficient structure, but it could be detrimental to optimizationperformance.
Revised: The split branch trick serves a memory-efficient structure, but it could be detrimental to optimizationperformance.

Original: Therefore, we implement the trunk-branch structure only for the sake of memoryefficiency of GPU.
Revised: Therefore, we implement the trunk-branch structure only for the sake of memoryefficiency of GPU.

Original: The baseline attention network structure includes dropout layers [51], and the dropout layers’ randomness makes the generator stochastic.
Revised: The baseline attention network structure includes dropout layers [51], and the dropout layers’ randomness makes the generator stochastic.

Original: A random input feed z is also a source of stochastic behavior.
Revised: A random input feed z is also a source of stochastic behavior.

Original: We experimented with both stochastic generators (Figure 5) and non-stochastic generators (Figure 4).
Revised: We experimented with both stochastic generators (Figure 5) and non-stochastic generators (Figure 4).

Original: However, for critic networks, we maintained the dropout layers as non-zero.
Revised: However, for critic networks, we maintained the dropout layers as non-zero.

Original: 3.4 A complementary strategy of generative network and evolution
Revised: 3.4 A complementary strategy of generative network and evolution

Original: Figure 1 shows the full algorithm of GEO.
Revised: Figure 1 shows the full algorithm of GEO.

Original: Training a surrogate model (critic network) is the essentialpart of a surrogate model-based optimizer, but it is also the trickiest part.
Revised: Training the surrogate model (critic network) is theessential part of a surrogate model-based optimizer, but it is also the trickiest part.

Original: The point is that the trainingdata (the true data in GAN concept) is not prepared, and the data can be only acquired throughon-the-fly searches.
Revised: The point is thatthe training data (the true data in GAN concepts) is not prepared, and it can be only acquired throughan on-the-fly search.

Original: Without prepared data, training can be unstable since outbreaks of new datamake the training region fluctuate.
Revised: Without prepared data, training can be unstable since outbreaks of new datamake the training region fluctuate.

Original: In this case, the algorithm diverges for the following reasons:1.
Revised: In this case, the algorithm diverges for the following reasons:

Original: The generator suggests an input variable x in a wrong direction.
Revised: The generator suggests an input variable x in a wrong direction.

Original: The critic network is trainedwith input variable x , but x has no information of a Pareto front.
Revised: The critic network is trained with input variable x , but x has no information of a Pareto front.

Original: The critic network trains thegenerator, but it does not give meaningful information.
Revised: The critic network trains the generator, but it does not give meaningful information.

Original: In short, the divergence is a result of evil cycles of two networks.
Revised: In short, the divergence is a result of evil cycles of two networks.

Original: The local sampling of L-GSO seems to be a simple stabilization method. In a case of N =1, L-GSOsamples data in  a local region, where the center is a current point.The current point is the Pareto-front,in this case.
Revised: The local sampling method in L-GSO can be a solution for stabilizing a surrogate network in asingle-objective function. It naturally creates a local trust region in the vicinity of an optimized point.

Original: However, it can be challenging in N -objective functions (in cases of N >1).
Revised: However, it can be challenging in N -objective functions.

Original: Since the Pareto-frontis R N − 1 surface (not a single point), there will be a lot of centers of sampling. Then, it is not localanymore.
Revised: Since the Pareto front is R n − 1 surface, itrequires R n − 1 dimensional sampling region, which cannot be local.

Original: The local sampling method cannot be used in multi-objective problems.
Revised: [Figure]

Original: Therefore, we need to devise a stabilizing method for multi-objective functions.
Revised: Therefore, we need to devise a stabilizing method for multi-objective functions.

Original: We suggest thatthe evolution strategy can be a good solution.
Revised: We suggest that theevolution strategy can be a good solution.

Original: The role of an evolution pool is to trap G andcorresponding x near the Pareto-front (rank 1). At the same time, the data that is far from the Pareto-front is discarded.
Revised: Only the optimized generators survive in the evolutionstrategy.

Original: Also, it slows down the fluctuation of the training data.
Revised: At the same time, it slows down the fluctuation of the pool.

Original: As a result, thetraining data region is stabilized around the Pareto-front.
Revised: As a result, the training dataregion becomes stabilized around the Pareto front.

Original: We expect the stabilization method to serve as an anchor, which prevents the training data fromfloating.
Revised: We expect the stabilized data region to serve as an anchor, which prevents the training data fromfloating.

Original: At the same time, a properly trained critic network provides better mutation strategies.
Revised: At the same time, a properly trained critic network provides backpropagation for effectivemutations of generators.

Original: Theinterdependent cooperation ofan evolution strategy and surrogate models is our main idea.
Revised: The interdependent cooperation of evolution strategy and surrogate model isour main idea.

Original:  Experimental results
Revised:  Experimental results

Original: In this section, we compare the black-box optimization results of GEO with baseline optimizers.
Revised: In this section, we compare the black-box optimization results of GEO with baseline optimizers.

Original: Wetested single and multi-objective functions, stochastic and non-stochastic functions.
Revised: Wetested single and multi-objective functions, stochastic and non-stochastic functions.

Original: The baselineoptimizers are Bayesian optimization (Gaussian process), NSGA-II [7] (GA in 1-object), NSGA-III[15], MOEA/D [18], and CMA-ES [9] evolution algorithms. 
Revised: The baselineoptimizers are Bayesian optimization (Gaussian process), NSGA-II (NSGA2) [7], and CMA-ESevolution algorithms. For the implementation of NSGA-II and CMA-ES, we used the Pymoo package[52].

Original: 4.1 Single objective functions
Revised: 4.1 Single objective functions

Original: [Figure] Figure 3a) shows performances of optimizers according to dimensions.
Revised: Figure 3a) shows performances of optimizers according to dimensions.

Original: For the single-objective testfunction, we used Styblinski-Tang function.
Revised: For the single-objective testfunction, we used Styblinski-Tang function.

Original: At dimension = 2 , Bayesian optimization shows thebest performance.
Revised: At dimension = 2 , Bayesian optimization shows thebest performance.

Original: However, as the dimension increases, GEO outperforms baseline optimizers.
Revised: However, as the dimension increases, GEO outperforms baseline optimizers.

Original: Atdimension = 8 , 192 , baseline optimizers rarely finds the global minimum, while the GEO showsbetter performances.
Revised: Atdimension = 8 , 192 , baseline optimizers rarely finds the global minimum, while the GEO showsbetter performances.

Original: To see how the depth of the generator network affects the performance, we also tested GEO with asingle-layer generator.
Revised: To see how the depth of the generator network affects the performance, we also tested GEO with asingle-layer generator.

Original: The single-layer generator appears to have little optimization capability.
Revised: The single-layer generator appears to have little optimization capability.

Original: Evenin the dimension = 2 problem, it shows a considerably slow optimization.
Revised: Evenin the dimension = 2 problem, it shows a considerably slow optimization.

Original: Bayesian optimization is a powerful method for high-cost black-box problems, but its performancecan be weakened when the problem requires numerous function calls.
Revised: Bayesian optimization is a powerful method for high-cost black-box problems, but its performancecan be weakened when the problem requires numerous function calls.

Original: The computational complexityof Bayesian optimization is known to be O ( n 3 ) [3].
Revised: The computational complexityof Bayesian optimization is known to be O ( n 3 ) [3].

Original: Figure 3b) shows the computational time of
Revised: Figure 3b) shows the computational time of

Revised: [Table]

Original: GEO, NSGA-II, and Bayesian optimization.
Revised: GEO, NSGA-II, and Bayesian optimization.

Original: Like most evolution strategies, GEO is designed to havea computational complexity of O ( n ) .
Revised: Like most evolution strategies, GEO is designed to havea computational complexity of O ( n ) .

Original: Figure 3c) shows performance comparisons to a Local Surrogate Model (LSM), a modification of the L-GSO algorithm.
Revised: Figure 3c) shows performance comparisons to a Local Surrogate Model (LSM), a modificationof the L-GSO algorithm.

Original: LSM follows the general outline of L-GSO GAN implementation but adopts theself-attention network used in GEO.
Revised: LSM follows the general outline of L-GSO GAN implementation butadopts the self-attention network used in GEO.

Revised: Finding an optimal hyperparameter set for all testfunctions is nearly impossible since the behavior of LSM greatly varies according to hyperparameters.

Revised: Therefore we investigate relative properties of LSM according to dimensions.

Original: In the Styblinski-Tang function, LSM shows worse performancethan in other functions.
Revised: In the Styblinski-Tangfunction, LSM shows worse performance than in other functions.

Original: Also, the performance of LSM rapidly decreases as the dimension increases.
Revised: Also, the performance of LSMrapidly decreases as the dimension increases.

Original: On the other hand, GEO shows better optimization performance under various conditions.
Revised: On the other hand, GEO shows better optimizationperformance under various conditions.

Original: 4.2 Multi objective functions
Revised: 4.2 Multi objective functions

Original: In this section, we show optimization performance comparisons in multi-objective problems.
Revised: In this section, we show optimization performance comparisons in multi-objective problems.

Original: Weonly compare GEO and evolution strategies because the high-dimensional problems require a lot offunction calls.
Revised: We onlycompare GEO and evolution strategies because the high-dimensional problems require immoderatefunction calls.

Original: Figure 4 shows optimization results in the non-stochastic-multi-objective functions.
Revised: Figure 4 shows optimization results in the non-stochastic-multi-objective functions.

Original: Each figureis a result after 100,000 function calls.
Revised: Each figure is a Pareto front plot after 100,000 function calls.

Original: For the multi-objective test functions, ZDT functions and [Table] combined functions F =
Revised: For the multi-objective test functions, ZDT functionsand combined functions F =

Original: [ f 1 , f 2 ] ( f i : Styblinski-Tang, Ackley and Rastrigin function) were used.
Revised: [ f 1 , f 2 ] ( f i : Styblinski-Tang, Ackley and Rastrigin function) wereused.

Original: A Latin Hyper Cube (LHC) [53] method is a good guess for initial states.
Revised: A Latin Hyper Cube (LHC) [53] method is a good guess for initial states.

Original: However, GEO cannotimplement LHC because it generates initial points x through the neural network.
Revised: However, GEOcannot implement LHC because it generates initial points x through the neural network.

Original: We gave two initialstates in the baseline optimizer to control performance according to the initial state.
Revised: We gave twoinitial states in the baseline optimizer to control performance according to the initial state.

Original: NSGA 2 - Iuses GEO’s initial distribution x ∈ G ( z ) as its initial points, while NSGA 2 - LHC uses LHC-initialpoints.
Revised: NSGA 2 - Iuses GEO’s initial distribution x ∈ G ( z ) as its initial points, while NSGA 2 - LHC uses LHC-initialpoints.

Original: In the high-dimension, GEO outperforms baseline optimizers.
Revised: In most cases, GEO outperforms baseline optimizers.

Original: The choice of initial points for NSGAII rarely affects final results.
Revised: The choice of initial points for NSGA-II rarelyaffects final results.

Original: A slightly different result appears in ZDT2.
Revised: A slightly different result appears in ZDT2.

Original: In ZDT2, GEO finds anoptimal point, but it fails to find a global shape of the Pareto-front.
Revised: In ZDT2, GEO finds an optimalpoint, but it fails to find a global shape of the Pareto front.

Original: Figure 6 shows the result of ZDT3optimization according to dimensions.

Original: Classical ES algorithms significantly reduce performance inhigh-dimensional space, while GEO shows more robust performance in high-dimensional space.

Original: Figure 5 shows optimization results in the stochastic-multi-objective functions.
Revised: Figure 5 shows optimization results in the stochastic-multi-objective functions.

Original: The optimization ofstochastic functions is defined as follows
Revised: The optimization ofstochastic functions is defined as follows

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: N is a random normal distribution, and we set µ = 0 . 0 and σ = 1 . 0 .
Revised: N is a random normal distribution, and we set µ = 0 . 0 and σ = 1 . 0 .

Revised: Figure 5 shows optimizationresult of stochastic-multi-objective functions.

Original: GEO outperforms baselineoptimizers even in a stochastic environment, but it still has a single-point collapsing problem in the ZDT2 function.
Revised: GEO outperforms baseline optimizers even in astochastic environment, but it still has a single-point-convergence problem in the ZDT2 function.

Original: 4.3 Cartpole-v1
Revised: 4.3 Cartpole-v1

Original: For the variety of test functions, we optimized the OpenAI [54] Cartpole-V1 by converting it into ablack-box problem.
Revised: For the variety of test functions, we optimized the OpenAI [54] Cartpole-V1 by converting it intoa black-box problem.

Original: It is also a simple toy model of time-sequential input-output (I/O) problems.
Revised: It is also a simple toy model of time-sequential I/O problems like electronicdevice simulators.

Original: Cartpole-V1 is a test package that is mainly used in reinforcement learning [34][35].
Revised: Cartpole-V1 is a test package that is mainly used in reinforcement learning [34][35].

Original: Reinforcementlearning requires a series of I/O structure.
Revised: In the reinforcement learning perspective, its action space is regarded as two-dimensional and time-sequential. Reinforcement learning is trained by continuously observing rewards and states for actions duringlearning.

Original: However, in the black-box problem, the entire inputsequence is assumed as one large input, and only the final score is measured without observing theintermediate rewards and states.
Revised: However, in the black-box problem, the entire input sequence is assumed as one large input,and only the final score is measured without observing the intermediate rewards and states.

Original: Since this experiment is a toy model of real-world problems whichhave stochastic environments, we kept the Cartpole-V1 stochastic.
Revised: [s = 0.999] 1 : [s = 0.959] 2 : [s = 0.999] 3 : [s = 0.999] 4 : [s = 0.999]

Original: The final score is measured in a relative score to the sequence length.
Revised: The final score is measured in a relative score to the sequence length.

Original: (If the Cartpole is alive form -length in n -sequences, the score is m/n .)
Revised: (If the Cartpole is alive form -length in n -sequences, the score is m/n .)

Original: Therefore, the maximum and minimum score set to 1 .
Revised: Therefore, the maximum and minimum score set to 1 .

Original: 0and 0 . 0 .
Revised: 0and 0 . 0 .

Original: GEO outperforms others from 256 to 2048 dimensions in the experiment (Table 1).
Revised: GEO outperforms NSGA-II from 32 to 512 dimensions in the experiment (Table 1).

Original: [s = 0.999] 1 : [s = 0.959] 2 : [s = 0.999] 3 : [s = 0.999] 4 : [s = 0.999] Cartpole-V1 isdifficult to solve with a black-box optimization approach due to the stochastic changeof the initial state, but it shows a clear performance difference between the black-box optimizers.
Revised: Since this experiment is a toy model of real-world problems which have stochastic environments, wekept the Cartpole stochastic.

Original: [Figure] We also optimized LeNet-5, which is trained with the MNIST dataset.
Revised: We also optimized LeNet-5, which is trained with the MNIST dataset.

Original: The optimization goal is togenerate an image that makes the LeNet-5 predict a target number with a maximum score (maximumprediction score = 1 . 0 ).
Revised: The optimization goal is togenerate an image that makes the LeNet-5 predict a target number with a maximum score (maximumprediction score = 1 . 0 ).

Original: The LeNet-5 is regarded as a non-differentiable black-box.
Revised: The LeNet-5 is regarded as a non-differentiable black-box.

Original: After 50 , 000function calls, the final scores of generated images reach very close to the maximum score (Figure 7).
Revised: After 50 , 000function calls, the final scores of generated images reach very close to the maximum score (Figure 6).

Original: In the related experiment, L-GSO, generative models appear to be better at finding the local optimumin sub-manifolds.
Revised: In the related experiment, L-GSO, generative models appear to be better at finding the local optimumin sub-manifolds.

Original: For the same reason, we expected to see the sub-manifold structure in thegeneratedimage, but the generated image does not seem intuitive.
Revised: We wanted to see the sub-manifold structure in the generated image, but it does notseem intuitive.

Original:  Discussion
Revised:  Discussion

Original: Often, a neural network’s learning mechanism is likened to learning manifolds in a high-dimensionalspace.
Revised: Often, a neural network’s learning mechanism is likened to learning manifolds in a high-dimensionalspace.

Original: Similarly, we guess that the critic network in GEO learns low-dimensional manifolds in thehigh-dimensional space.
Revised: Similarly, we guess that the critic network in GEO learns low-dimensional manifolds in thehigh-dimensional space.

Original: Therefore, we expect that finding global or local optima would be easy ifthe optima are in low-dimensional manifolds.
Revised: Therefore, we expect that finding global or local optima would be easy ifthe optima are in low-dimensional manifolds.

Original: Also, we consider that it is the reason why the depth ofgenerators is important. Meanwhile, we guess that the collapse problem of ZDT2 is caused by a concave shape of its Paretofront.
Revised: Also, we consider that it is the reason why GEO withsingle-layer generators optimizes a target very slowly. Stabilization of critic network training is another key.

Original: This is because, when the data is formed as a concave shape, a non-dominated sorting selectsthe edge state first. We have yet to find a clear solution to solve the collapse problem withoutcompromising performance.
Revised: Although we implemented the evolutionstrategy in this study, any other methods which can stabilize the training process would be helpful.

Original:  Conclusion
Revised:  Conclusion

Original: We have described a method for stochastic-multi-objective black-box optimization.
Revised: We have described a method for stochastic-multi-objective black-box optimization.

Original: GEO is aninterdependent cooperation method of generative neural networks andthe evolution strategy.
Revised: GEO is aninterdependent cooperation method of generative neural networks and evolution strategy.

Original: Theevolution strategy provides a stable training region for critic networks, and the critic networks provideefficient mutations to the evolution strategy.
Revised: Theevolution strategy provides a stable training region for critic networks, and the critic networks provideefficient mutations to the evolution strategy.

Original: As our design intent, GEO seems to work appropriatelyin stochastic and high-dimensional multi-objective test functions.
Revised: As our design intent, GEO seems to work appropriatelyin stochastic and high-dimensional multi-objective test functions.

Revised: It can optimize not only generaltest functions but also sequential type and image type black-boxes.

Revised: We presume that these resultscome from the manifold searching ability of neural networks.

Original: Meanwhile, the Pareto-front collapsing problem, shown in ZDT2, is an important issue to be dealtwith.
Revised: Meanwhile, the Pareto front collapsingproblem, shown in ZDT2, is an important issue to be dealt with.

Original: Another limitation of GEO is the GPU memory consumption problem.
Revised: Another limitation of GEO is the GPU memory consumption problem.

Original: The excessive memoryconsumption of attention networks limits its search space to around 10 , 000 dimensions.
Revised: Excessive memory consumption of attention networks limits itssearch space to around 10 , 000 dimensions.

Original: In futureresearches, we can study other memory-efficient networks to solve this problem.
Revised: In future research, we can study other memory-efficientnetworks to solve this problem.

Original: GEO is designed for optimization in extremely high-dimensions.

Original: However, the performance at lowerdimensions is not guaranteed (see supplement). We think that [1,000 < d < 10,000] is the practicalrange of use of GEO, unless we improve the efficiency of network structures. In addition, the mutationof generators concentrates on an exploit, the explore strategy could be weak. In the next study, astrong exploit & explore strategy should be added to improve optimization performance.
Revised: We hope that this study can contribute to various black-box optimization researches in science andtechnology.

Original: Checklist
Revised: Checklist

Original: For all authors...
Revised: For all authors...

Original: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’scontributions and scope?
Revised: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’scontributions and scope?

Original: [Yes]
Revised: [Yes]

Original: (b) Did you describe the limitations of your work?
Revised: (b) Did you describe the limitations of your work?

Original: [Yes] See discussion and conclusion(c)
Revised: [Yes] See discussion and conclusion(c)

Original: Did you discuss any potential negative societal impacts of your work?
Revised: Did you discuss any potential negative societal impacts of your work?

Original: [Yes] (d) Have you read the ethics review guidelines and ensured that your paper conforms to
Revised: [Yes](d) Have you read the ethics review guidelines and ensured that your paper conforms to

Original: them?
Revised: them?

Original: [Yes]
Revised: [Yes]

Original: If you are including theoretical results...
Revised: If you are including theoretical results...

Original: (a) Did you state the full set of assumptions of all theoretical results?
Revised: (a) Did you state the full set of assumptions of all theoretical results?

Original: [N/A]
Revised: [N/A]

Original: (b) Did you include complete proofs of all theoretical results?
Revised: (b) Did you include complete proofs of all theoretical results?

Original: [N/A]
Revised: [N/A]

Original: If you ran experiments...
Revised: If you ran experiments...

Original: (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
Revised: (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?

Original: [Yes] See supplementary for implementation details
Revised: [Yes] See supplementary for implementation details

Original: (b) Did you specify all the training details (e g., data splits, hyperparameters, how theywere chosen)?
Revised: (b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?

Original: [Yes] See supplementary for implementation details
Revised: [Yes] See supplementary for implementation details

Original: (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
Revised: (c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)?

Original: [Yes] See supplementary for experimental details
Revised: [Yes] See supplementary for experimental details

Original: (d) Did you include the total amount of compute and the type of resources used (e g., typeof GPUs, internal cluster, or cloud provider)?
Revised: (d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?

Original: [Yes] See supplementary for experimentaldetails
Revised: [Yes] See supplementary for experimental details

Original: If you are using existing assets (e g., code, data, models) or curating/releasing new assets...
Revised: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...

Original: (a) If your work uses existing assets, did you cite the creators?
Revised: (a) If your work uses existing assets, did you cite the creators?

Original: [Yes] See references
Revised: [Yes] See references

Original: (b) Did you mention the license of the assets?
Revised: (b) Did you mention the license of the assets?

Original: [Yes] See supplementary (c)
Revised: [Yes] See supplementary(c)

Original: Did you include any new assets either in the supplemental material or as a URL
Revised: Did you include any new assets either in the supplemental material or as a URL?

Original: [No]
Revised: [No]

Original: (d) Did

Original: you discuss whether

Original: and

Original:  how consent was obtained from people whose data you’re
Revised: (d) Did you discuss whether and how consent was obtained from people whose data you’reusing/curating?

Original: using

Original: /curating?

Original: [Yes
Revised: [Yes]

Original: (e) Did you discuss whether the data you are using/curating contains personally identifiable
Revised: (e) Did you discuss whether the data you are using/curating contains personally identifiable

Original: information or offensive content?
Revised: information or offensive content?

Original: [Yes
Revised: [Yes]

Original: If you used crowdsourcing or conducted research with human subjects...
Revised: If you used crowdsourcing or conducted research with human subjects...

Original: (a) Did you include the full text of instructions given to participants and screenshots, ifapplicable?
Revised: (a) Did you include the full text of instructions given to participants and screenshots, if applicable?

Original: [N/A]
Revised: [N/A]

Original: [Table] (b) Did you describe any potential participant risks, with links to Institutional Review[Table]
Revised: (b) Did you describe any potential participant risks, with links to Institutional Review

Original: Board (IRB) approvals, if applicable?
Revised: Board (IRB) approvals, if applicable?

Original: [N/A]
Revised: [N/A

Original: [Table] (c) Did you include the estimated hourly wage paid to participants and the total amount [Table] spent on participant compensation?
Revised: ](c) Did you include the estimated hourly wage paid to participants and the total amountspent on participant compensation?

Original: [N/A]
Revised: [N/A]

Original: [Table]