Original: B AYESIAN L EARNING TO O PTIMIZE : Q UANTIFYING THE O PTIMIZER U NCERTAINTY
Revised: B AYESIAN L EARNING TO O PTIMIZE : Q UANTIFYING THE O PTIMIZER U NCERTAINTY

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Optimizing an objective function with uncertainty awareness is well-known to improve the accuracy and conﬁdence of optimization solutions.
Revised: Optimizing an objective function with uncertainty awareness is well-known to improve the accuracy and conﬁdence of optimization solutions.

Original: Meanwhile, another relevant but very different question remains yet open: how to model and quantify the uncertainty of an optimization algorithm itself?
Revised: Meanwhile, another relevant but very different question remains yet open: how to model and quantify the uncertainty of an optimization algorithm itself?

Original: To close such a gap, the prerequisite is to consider the optimizers as sampled from a distribution, rather than a few pre-deﬁned and ﬁxed update rules.
Revised: To close such a gap, the prerequisite is to consider the optimizers as sampled from a distribution, rather than a few pre-deﬁned and ﬁxed update rules.

Original: We ﬁrst take the novel angle to consider the algorithmic space of optimizers, each being parameterized by a neural network.
Revised: We ﬁrst take the novel angle to consider the algorithmic space of optimizers, each being parameterized by a neural network.

Original: We then propose a Boltzmann-shaped posterior over this optimizer space, and approximate the posterior locally as Gaussian distributions through variational inference.
Revised: We then propose a Boltzmann-shaped posterior over this optimizer space, and approximate the posterior locally as Gaussian distributions through variational inference.

Original: Our novel model, Bayesian learning to optimize ( BL2O ) is the ﬁrst study to recognize and quantify the uncertainty of the optimization algorithm.
Revised: Our novel model, Bayesian learning to optimize ( BL2O ) is the ﬁrst study to recognize and quantify the uncertainty of the optimization algorithm.

Original: Our experiments on optimizing test functions, energy functions in proteinprotein interactions and loss functions in image classiﬁcation and data privacy attack demonstrate that, compared to state-of-the-art methods, BL2O improves optimization and uncertainty quantiﬁcation (UQ) in aforementioned problems as well as calibration and out-of-domain detection in image classiﬁcation.
Revised: Our experiments on optimizing test functions, energy functions in proteinprotein interactions and loss functions in image classiﬁcation and data privacy attack demonstrate that, compared to state-of-the-art methods, BL2O improves optimization and uncertainty quantiﬁcation (UQ) in aforementioned problems as well as calibration and out-of-domain detection in image classiﬁcation.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Computational models of many real-world applications involve optimizing non-convex objective functions.
Revised: Computational models of many real-world applications involve optimizing non-convex objective functions.

Original: As the non-convex optimization problem is NP-hard, no optimization algorithm (or optimizer) could guarantee the global optima in general, and instead, their solutions’ usefulness (sometimes based on their proximity to the optima), when the optima are unknown, can be very uncertain.
Revised: As the non-convex optimization problem is NP-hard, no optimization algorithm (or optimizer) could guarantee the global optima in general, and instead, their solutions’ usefulness (sometimes based on their proximity to the optima), when the optima are unknown, can be very uncertain.

Original: Being able to quantify such uncertainty is important to not only assessing the solution uncertainty after optimization but also enhancing the search efﬁciency during optimization.
Revised: Being able to quantify such uncertainty is important to not only assessing the solution uncertainty after optimization but also enhancing the search efﬁciency during optimization.

Original: For instance, reliable and trustworthy machine learning models demand uncertainty awareness and quantiﬁcation during training (optimizing) such models, whereas in reality deep neural networks without proper modeling of uncertainty suffer from overconﬁdence and miscalibration (Guo et al., 2017).
Revised: For instance, reliable and trustworthy machine learning models demand uncertainty awareness and quantiﬁcation during training (optimizing) such models, whereas in reality deep neural networks without proper modeling of uncertainty suffer from overconﬁdence and miscalibration (Guo et al., 2017).

Original: In another application example of protein docking, although there exists epistemic uncertainty of the objective function and the aleatoric uncertainty of the protein structure data (Cao & Shen, 2020), state-ofthe-art methods only predict several single solutions (Porter et al., 2019) without any associated uncertainty, which makes those predictions hard to be interpreted by the end users.
Revised: In another application example of protein docking, although there exists epistemic uncertainty of the objective function and the aleatoric uncertainty of the protein structure data (Cao & Shen, 2020), state-ofthe-art methods only predict several single solutions (Porter et al., 2019) without any associated uncertainty, which makes those predictions hard to be interpreted by the end users.

Original: Various optimization methods have been proposed in response to the need of uncertainty awareness.
Revised: Various optimization methods have been proposed in response to the need of uncertainty awareness.

Original: Stochastic optimization methods like random search (Zhigljavsky, 2012), simulated annealing (Kirkpatrick et al., 1983), genetic algorithms (Goldenberg, 1989) and particle swarm optimization (Kennedy & Eberhart, 1995) injected the randomness into the algorithms in order to reduce uncertainties.
Revised: Stochastic optimization methods like random search (Zhigljavsky, 2012), simulated annealing (Kirkpatrick et al., 1983), genetic algorithms (Goldenberg, 1989) and particle swarm optimization (Kennedy & Eberhart, 1995) injected the randomness into the algorithms in order to reduce uncertainties.

Original: However, these methods do not provide the uncertainty quantiﬁcation (UQ) of solutions.
Revised: However, these methods do not provide the uncertainty quantiﬁcation (UQ) of solutions.

Original: Recently, there have been growing interests in applying inference-based methods to optimization problems (Brochu et al., 2010; Shapiro, 2000; Pelikan et al., 1999).
Revised: Recently, there have been growing interests in applying inference-based methods to optimization problems (Brochu et al., 2010; Shapiro, 2000; Pelikan et al., 1999).

Original: Generally, they transfer the uncertainties within the data and model into the ﬁnal solution by modelling the posterior distribution over the global optima.
Revised: Generally, they transfer the uncertainties within the data and model into the ﬁnal solution by modelling the posterior distribution over the global optima.

Original: For instance, Bijl et al.
Revised: For instance, Bijl et al.

Original: (2016) uses sequential Monte Carlo to approximate the distribution over the optima with Thompson sampling as the search strategy.
Revised: (2016) uses sequential Monte Carlo to approximate the distribution over the optima with Thompson sampling as the search strategy.

Original: Hern´andez-Lobato et al.
Revised: Hern´andez-Lobato et al.

Original: (2014) uses kernel approximation for modelling the posterior over the optimum under Gaussian process.
Revised: (2014) uses kernel approximation for modelling the posterior over the optimum under Gaussian process.

Original: Ortega et al.
Revised: Ortega et al.

Original: ; Cao & Shen (2020) directly model the posterior over the optimum as a Boltzmann distribution.
Revised: ; Cao & Shen (2020) directly model the posterior over the optimum as a Boltzmann distribution.

Original: They not only surpass the previous methods in accuracy and efﬁciency, but also provide easy-to-interpret uncertainty quantiﬁcation.
Revised: They not only surpass the previous methods in accuracy and efﬁciency, but also provide easy-to-interpret uncertainty quantiﬁcation.

Original: Despite progress in optimization with uncertainty-awareness, signiﬁcant open questions remain.
Revised: Despite progress in optimization with uncertainty-awareness, signiﬁcant open questions remain.

Original: Existing methods consider uncertainty either within the data or the model (including objective functions)
Revised: Existing methods consider uncertainty either within the data or the model (including objective functions)

Original: (Kendall & Gal, 2017; Ortega et al., 2012; Cao & Shen, 2020).
Revised: (Kendall & Gal, 2017; Ortega et al., 2012; Cao & Shen, 2020).

Original: However, no attention was ever paid to the uncertainty arising from the optimizer that is directly responsible for deriving the end solutions with given data and models.
Revised: However, no attention was ever paid to the uncertainty arising from the optimizer that is directly responsible for deriving the end solutions with given data and models.

Original: The optimizer is usually pre-deﬁned and ﬁxed in the optimization algorithm space.
Revised: The optimizer is usually pre-deﬁned and ﬁxed in the optimization algorithm space.

Original: For instance, there are several popular update rules in Bayesian optimization, such as expected improvement Vazquez & Bect (2010) or upper conﬁdence bound Srinivas et al.
Revised: For instance, there are several popular update rules in Bayesian optimization, such as expected improvement Vazquez & Bect (2010) or upper conﬁdence bound Srinivas et al.

Original: (2009), that are chosen and ﬁxed for the entire process.
Revised: (2009), that are chosen and ﬁxed for the entire process.

Original: For Bayesian neural networks training, the update rule is usually chosen off-the-shelf, such as Adam, SGD, or RMSDrop.
Revised: For Bayesian neural networks training, the update rule is usually chosen off-the-shelf, such as Adam, SGD, or RMSDrop.

Original: The uncertainty in the optimizer is intrinsically deﬁned over the optimizer space and important to the optimization and UQ solutions.
Revised: The uncertainty in the optimizer is intrinsically deﬁned over the optimizer space and important to the optimization and UQ solutions.

Original: However, such uncertainty is unwittingly ignored when the optimizer is treated as a ﬁxed sample in the space.
Revised: However, such uncertainty is unwittingly ignored when the optimizer is treated as a ﬁxed sample in the space.

Original: To ﬁll the aforementioned gap, the core intellectual value of this work is to recognize and quantify a new form of uncertainty, that lies in the optimization algorithm (optimizer), besides the classical data- or model- based uncertainties (also known as epistemic and aleatoric uncertainties).
Revised: To ﬁll the aforementioned gap, the core intellectual value of this work is to recognize and quantify a new form of uncertainty, that lies in the optimization algorithm (optimizer), besides the classical data- or model- based uncertainties (also known as epistemic and aleatoric uncertainties).

Original: The underlying innovation is to treat an optimizer as a random sample from the algorithmic space , rather than one of a few hand-crafted update rules.
Revised: The underlying innovation is to treat an optimizer as a random sample from the algorithmic space , rather than one of a few hand-crafted update rules.

Original: The key enabling technique is to consider the algorithmic space being parameterized by a neural network.
Revised: The key enabling technique is to consider the algorithmic space being parameterized by a neural network.

Original: We then leverage a Boltzmann-shaped posterior over the optimizers, and approximate the posterior locally as Gaussian distributions through variational inference.
Revised: We then leverage a Boltzmann-shaped posterior over the optimizers, and approximate the posterior locally as Gaussian distributions through variational inference.

Original: Our approach, Bayesian learning to optimize ( BL2O ), for the ﬁrst time addresses the modeling of the optimizer-based uncertainty.
Revised: Our approach, Bayesian learning to optimize ( BL2O ), for the ﬁrst time addresses the modeling of the optimizer-based uncertainty.

Original: Extensive experiments on optimizing test functions, energy functions in a bioinformatics application, and loss functions in the image classiﬁcation and data privacy attack demonstrate that compared to the start-of-art methods, BL2O substantially improves the performance of optimization and uncertainty quantiﬁcation, as well as calibration and out-of-domain detection in classiﬁcation.
Revised: Extensive experiments on optimizing test functions, energy functions in a bioinformatics application, and loss functions in the image classiﬁcation and data privacy attack demonstrate that compared to the start-of-art methods, BL2O substantially improves the performance of optimization and uncertainty quantiﬁcation, as well as calibration and out-of-domain detection in classiﬁcation.

Original: In the following sections, we ﬁrst review related methods in details and reveal the remaining gap.
Revised: In the following sections, we ﬁrst review related methods in details and reveal the remaining gap.

Original: We then formally deﬁne the problem of optimization with uncertainty quantiﬁcation and point out the optimizer as a source of uncertainty.
Revised: We then formally deﬁne the problem of optimization with uncertainty quantiﬁcation and point out the optimizer as a source of uncertainty.

Original: After formally deﬁning the optimizer space, the optimal optimizer as a random vector in the space, and the optimizer uncertainty, we propose our novel model, BL2O .
Revised: After formally deﬁning the optimizer space, the optimal optimizer as a random vector in the space, and the optimizer uncertainty, we propose our novel model, BL2O .

Original: And lastly, we compare our BL2O with both Bayesian and non-Bayesian competing methods on extensive test functions and real-world applications.
Revised: And lastly, we compare our BL2O with both Bayesian and non-Bayesian competing methods on extensive test functions and real-world applications.

Original:  R ELATED WORK
Revised:  R ELATED WORK

Original: Many works (Wang & Jegelka, 2017; Hennig & Schuler, 2012) studied optimization with uncertainty quantiﬁcation under the framework of Bayesian optimization (Shahriari et al., 2016; Brochu et al., 2010).
Revised: Many works (Wang & Jegelka, 2017; Hennig & Schuler, 2012) studied optimization with uncertainty quantiﬁcation under the framework of Bayesian optimization (Shahriari et al., 2016; Brochu et al., 2010).

Original: In these studies, multiple objectives are sampled from the posterior over the objectives ( p ( f |D ) ), where D is the observed data.
Revised: In these studies, multiple objectives are sampled from the posterior over the objectives ( p ( f |D ) ), where D is the observed data.

Original: Each sampled objective is optimized for obtaining samples of the global optima: w ∗ so that the empirical distribution over w ∗ can be built.
Revised: Each sampled objective is optimized for obtaining samples of the global optima: w ∗ so that the empirical distribution over w ∗ can be built.

Original: Approximation is much needed since those approaches need optimization for every sample.
Revised: Approximation is much needed since those approaches need optimization for every sample.

Original: For instance,
Revised: For instance,

Original: Henr´andez-Lobato et al.
Revised: Henr´andez-Lobato et al.

Original: (2014) uses kernel approximation to approximate the posterior distirbution.
Revised: (2014) uses kernel approximation to approximate the posterior distirbution.

Original: Another line of work uses various sampling schemes for estimating the density of posterior distributions.
Revised: Another line of work uses various sampling schemes for estimating the density of posterior distributions.

Original: For instance, Bijl et al.
Revised: For instance, Bijl et al.

Original: (2016) uses sequential Monte Carlo sampling.
Revised: (2016) uses sequential Monte Carlo sampling.

Original: De Bonet et al.
Revised: De Bonet et al.

Original: (1997) designs a randomized optimization algorithm that directly samples global optima.
Revised: (1997) designs a randomized optimization algorithm that directly samples global optima.

Original: These methods are much more efﬁcient, but their performance heavily depends on the objective landscapes.
Revised: These methods are much more efﬁcient, but their performance heavily depends on the objective landscapes.

Original: Moreover, a few studies (Ahmed et al., 2016; Lizotte, 2008; Osborne et al., 2009; Wu et al., 2017) in
Revised: Moreover, a few studies (Ahmed et al., 2016; Lizotte, 2008; Osborne et al., 2009; Wu et al., 2017) in

Original: Bayesian optimization utilize ﬁrst-order information to boost the performance of optimization.
Revised: Bayesian optimization utilize ﬁrst-order information to boost the performance of optimization.

Original: For instance, Osborne et al.
Revised: For instance, Osborne et al.

Original: (2009) uses gradient information to improve the covariance matrix in Gaussian process.
Revised: (2009) uses gradient information to improve the covariance matrix in Gaussian process.

Original: Wu et al.
Revised: Wu et al.

Original: (2017) embeds the derivative knowledge into the acquisition function which is optimized in every iteration.
Revised: (2017) embeds the derivative knowledge into the acquisition function which is optimized in every iteration.

Original: Finally, there are approaches (Ortega et al., 2012; Cao & Shen, 2020) that directly model the shape of posterior as the Boltzmann distributions: p ( w ∗ | D ) ∝ exp( − αf ( w ∗ )) , where α is the scheduled temperature constant.
Revised: Finally, there are approaches (Ortega et al., 2012; Cao & Shen, 2020) that directly model the shape of posterior as the Boltzmann distributions: p ( w ∗ | D ) ∝ exp( − αf ( w ∗ )) , where α is the scheduled temperature constant.

Original: They automatically adjust α during the search in order to balance the exploration-exploitation tradeoff.
Revised: They automatically adjust α during the search in order to balance the exploration-exploitation tradeoff.

Original: They beat previous work in terms of both efﬁciency and accuracy.
Revised: They beat previous work in terms of both efﬁciency and accuracy.

Original: However, as revealed earlier in the Introduction, none of the methods above consider the uncertainty within the optimizer.
Revised: However, as revealed earlier in the Introduction, none of the methods above consider the uncertainty within the optimizer.

Original:  M ETHODS
Revised:  M ETHODS

Original: Notation .
Revised: Notation .

Original: We use a bold-faced uppercase letter to denote a matrix (e.g. W ), a bold-faced lowercase letter to denote a vector (e.g. w ), and a normal lowercase letter to denote a scalar (e.g. w ).
Revised: We use a bold-faced uppercase letter to denote a matrix (e.g. W ), a bold-faced lowercase letter to denote a vector (e.g. w ), and a normal lowercase letter to denote a scalar (e.g. w ).

Original: 3. P ROBLEM S TATEMENT
Revised: 3. P ROBLEM S TATEMENT

Original: The goal of optimization is to ﬁnd the global optimum for an objective function f ( w ) w.r.t.
Revised: The goal of optimization is to ﬁnd the global optimum for an objective function f ( w ) w.r.t.

Original: w :
Revised: w :

Original: [Equation]
Revised: [Equation]

Original: w ∗ is assumed unknown and treated as a random vector in this study.
Revised: w ∗ is assumed unknown and treated as a random vector in this study.

Original: Once a optimizer obtains ˆ w , its estimate of w ∗ , it is important to assess the quality and the uncertainty of the solution.
Revised: Once a optimizer obtains ˆ w , its estimate of w ∗ , it is important to assess the quality and the uncertainty of the solution.

Original: Considering that many real-world objective functions are nonconvex and noisy in ˆ w , solution quality is often measured by || ˆ w − w ∗
Revised: Considering that many real-world objective functions are nonconvex and noisy in ˆ w , solution quality is often measured by || ˆ w − w ∗

Original: || , the proximity to the global optimum rather than that to the optimal function value.
Revised: || , the proximity to the global optimum rather than that to the optimal function value.

Original: Examples include energy functions as the objective and RMSDs as the proximity measure in protein docking (Lensink et al., 2007).
Revised: Examples include energy functions as the objective and RMSDs as the proximity measure in protein docking (Lensink et al., 2007).

Original: Therefore, the goal of uncertainty quantiﬁcation (UQ) here is the following:
Revised: Therefore, the goal of uncertainty quantiﬁcation (UQ) here is the following:

Original: [Equation]
Revised: [Equation]

Original: where r σ is the upper bound of || ˆ w − w ∗
Revised: where r σ is the upper bound of || ˆ w − w ∗

Original: || at σ conﬁdence level, and D denotes samples during optimization.
Revised: || at σ conﬁdence level, and D denotes samples during optimization.

Original: Such UQ results additionally provide conﬁdence in the solution ˆ w and improve model reliability for end users.
Revised: Such UQ results additionally provide conﬁdence in the solution ˆ w and improve model reliability for end users.

Original: To calculate the probability deﬁned in Eq 2 and perform UQ, a direct albeit challenging way is to model the posterior over w ∗ ( p ( w ∗ |D ) ) and then sample from the posterior.
Revised: To calculate the probability deﬁned in Eq 2 and perform UQ, a direct albeit challenging way is to model the posterior over w ∗ ( p ( w ∗ |D ) ) and then sample from the posterior.

Original: When the optimizer g is regarded ﬁxed in existing literature, the posterior is actually p ( w ∗ |D , g ) .
Revised: When the optimizer g is regarded ﬁxed in existing literature, the posterior is actually p ( w ∗ |D , g ) .

Original: A central contribution of ours is to further consider the optimizer as a source of uncertainty, model it as a random vector in an optimizer space, and perform posterior estimation of p ( w ∗ |D ) .
Revised: A central contribution of ours is to further consider the optimizer as a source of uncertainty, model it as a random vector in an optimizer space, and perform posterior estimation of p ( w ∗ |D ) .

Original: 3. O PTIMIZER U NCERTAINTY : A F RAMEWORK
Revised: 3. O PTIMIZER U NCERTAINTY : A F RAMEWORK

Original: An optimizer is directly responsible for optimization and thus naturally a source of solution uncertainty.
Revised: An optimizer is directly responsible for optimization and thus naturally a source of solution uncertainty.

Original: To address this often-neglected uncertainty source, we ﬁrst deﬁne the space of optimizers and then model an optimizer as a point in this space.
Revised: To address this often-neglected uncertainty source, we ﬁrst deﬁne the space of optimizers and then model an optimizer as a point in this space.

Original: Considering that many widely-used optimizers are iterative and using ﬁrst-order derivatives, we restrict the optimizer space as follows:
Revised: Considering that many widely-used optimizers are iterative and using ﬁrst-order derivatives, we restrict the optimizer space as follows:

Original: Deﬁnition 3.1 ((First-order Iterative) Optimizer Space)
Revised: Deﬁnition 3.1 ((First-order Iterative) Optimizer Space)

Original: We deﬁne a ﬁrst-order, iterative algorithmic space G , where each point g ∈ G is an iterative optimizer, that has the following mapping: g ( {∇ f ( w τ ) } tτ =1 ) =
Revised: We deﬁne a ﬁrst-order, iterative algorithmic space G , where each point g ∈ G is an iterative optimizer, that has the following mapping: g ( {∇ f ( w τ ) } tτ =1 ) =

Original: δ w t , where ∇ f ( w ) τ and δ w t are the gradient and the update vector at τ th and t th iteration, respectively.
Revised: δ w t , where ∇ f ( w ) τ and δ w t are the gradient and the update vector at τ th and t th iteration, respectively.

Original: Here we use g ( · ) to denote a pre-deﬁned update rule and the resulting optimizer.
Revised: Here we use g ( · ) to denote a pre-deﬁned update rule and the resulting optimizer.

Original: For instance, in gradient descent, g ( {∇ f ( w τ ) } tτ =1 ) = − α ∇ f ( w t ) , where α is the step size.
Revised: For instance, in gradient descent, g ( {∇ f ( w τ ) } t τ =1 ) = − α ∇ f ( w t ) , where α is the step size.

Original: Now that the optimizer space is deﬁned, we next deﬁne the (unknown) optimal optimizer and its uncertainty.
Revised: Now that the optimizer space is deﬁned, we next deﬁne the (unknown) optimal optimizer and its uncertainty.

Original: Deﬁnition 3.2 (Optimal Optimizer)
Revised: Deﬁnition 3.2 (Optimal Optimizer)

Original: We deﬁne the optimal optimizer g ∗ ∈ G as the optimizer that can obtain the lowest function value with a ﬁxed budget T :
Revised: We deﬁne the optimal optimizer g ∗ ∈ G as the optimizer that can obtain the lowest function value with a ﬁxed budget T :

Original: [Equation]
Revised: [Equation]

Original: where w tg =
Revised: where w tg =

Original: w t − 1 g + g ( {∇ f ( w τg ) } t − 1 τ =1 ) is the parameter value at t th iteration updated through the optimizer g .
Revised: w t − 1 g + g ( {∇ f ( w τg ) } t − 1 τ =1 ) is the parameter value at t th iteration updated through the optimizer g .

Original: In practice the optimal optimizer g ∗ is unknown so we treat g ∗ as a random vector and formally deﬁne the optimizer uncertainty as follows:
Revised: In practice the optimal optimizer g ∗ is unknown so we treat g ∗ as a random vector and formally deﬁne the optimizer uncertainty as follows:

Original: Deﬁnition 3.3 (Optimizer Uncertainty) Let G be the algorithmic space, where each point g ∈ G is an optimizer.
Revised: Deﬁnition 3.3 (Optimizer Uncertainty) Let G be the algorithmic space, where each point g ∈ G is an optimizer.

Original: We assume there is a prior distribution over the optimal optimizer g ∗ as p ( g ∗ ) .
Revised: We assume there is a prior distribution over the optimal optimizer g ∗ as p ( g ∗ ) .

Original: We also assume a likelihood distribution as p ( D| g ∗ ) , where D are the observed data (sample trajectory) given g ∗ .
Revised: We also assume a likelihood distribution as p ( D| g ∗ ) , where D are the observed data (sample trajectory) given g ∗ .

Original: Then we deﬁne the optimizer uncertainty through p ( g ∗ |D )
Revised: Then we deﬁne the optimizer uncertainty through p ( g ∗ |D )

Original: ∝ p ( D| g ∗ ) p ( g ∗ ) .
Revised: ∝ p ( D| g ∗ ) p ( g ∗ ) .

Original: To inject the optimizer uncertainty into p ( w ∗ |D ) , it is straightforward to have the following integration for posterior estimation:
Revised: To inject the optimizer uncertainty into p ( w ∗ |D ) , it is straightforward to have the following integration for posterior estimation:

Original: [Equation]
Revised: [Equation]

Original: 3. P ARAMETERIZING THE O PTIMIZER S PACE
Revised: 3. P ARAMETERIZING THE O PTIMIZER S PACE

Original: The optimizer uncertainty p ( g ∗ |D ) as deﬁned in Def. 3.3 can be intractable when there is no proper parameterization of the optimizer space G .
Revised: The optimizer uncertainty p ( g ∗ |D ) as deﬁned in Def. 3.3 can be intractable when there is no proper parameterization of the optimizer space G .

Original: Therefore, we next introduce possible ways of parameterizing G as deﬁned in Def. 3.1.
Revised: Therefore, we next introduce possible ways of parameterizing G as deﬁned in Def. 3.1.

Original: Parameterization through Hyperparameters of Speciﬁc Optimizers.
Revised: Parameterization through Hyperparameters of Speciﬁc Optimizers.

Original: A simple way to parameterize the optimizer space for classical optimizers (e.g. Gradient Descent, Adam) is based on their hyperparameters: G = H , where H is the hyperparameter space.
Revised: A simple way to parameterize the optimizer space for classical optimizers (e.g. Gradient Descent, Adam) is based on their hyperparameters: G = H , where H is the hyperparameter space.

Original: For instance, for gradient descent, we have H =
Revised: For instance, for gradient descent, we have H =

Original: ( α ) , where α is the learning rate.
Revised: ( α ) , where α is the learning rate.

Original: For Adam, we have H =
Revised: For Adam, we have H =

Original: ( α, β 1 , β 2 ) , where β 1 and β 2 are the coefﬁcients used for computing running averages of gradient and its square.
Revised: ( α, β 1 , β 2 ) , where β 1 and β 2 are the coefﬁcients used for computing running averages of gradient and its square.

Original: However, such parameterization has signiﬁcant drawbacks.
Revised: However, such parameterization has signiﬁcant drawbacks.

Original: The resulting algorithmic space G is very limited and heavily depends on the speciﬁc optimizer.
Revised: The resulting algorithmic space G is very limited and heavily depends on the speciﬁc optimizer.

Original: The G (a 1D space) parameterized by the hyperparameters of gradient descent is different from that (a 3D space) parameterized by the hyperparameters of Adam.
Revised: The G (a 1D space) parameterized by the hyperparameters of gradient descent is different from that (a 3D space) parameterized by the hyperparameters of Adam.

Original: In fact, each is a rather restricted region of the actual G .
Revised: In fact, each is a rather restricted region of the actual G .

Original: The intrinsic ﬂexibility (uncertainty) that lies in an iterative optimizer’s update rule is not explored at all in this parameterization.
Revised: The intrinsic ﬂexibility (uncertainty) that lies in an iterative optimizer’s update rule is not explored at all in this parameterization.

Original: These drawbacks are empirically demonstrated in Sec. 4.
Revised: These drawbacks are empirically demonstrated in Sec. 4.

Original: Parameterization through Neural Networks.
Revised: Parameterization through Neural Networks.

Original: In order to reasonably and accurately model the intrinsic uncertainty within the update rule, we need to ﬁnd a much more ﬂexible way for modelling g .
Revised: In order to reasonably and accurately model the intrinsic uncertainty within the update rule, we need to ﬁnd a much more ﬂexible way for modelling g .

Original: We thus consider to parameterize the optimizer space as a neural network: G = Θ , where each θ ∈ Θ are the parameters in the neural network.
Revised: We thus consider to parameterize the optimizer space as a neural network: G = Θ , where each θ ∈ Θ are the parameters in the neural network.

Original: Overcoming drawbacks of the optimizer space H by hyperparameters, Θ by neural network parameters generalizes update rules through neural networks that can represent a wide variety of functions.
Revised: Overcoming drawbacks of the optimizer space H by hyperparameters, Θ by neural network parameters generalizes update rules through neural networks that can represent a wide variety of functions.

Original: We note that this is also the space of meta-optimizers that learn to optimize (L2O) iterative update rules from data on a given task (Andrychowicz et al., 2016;
Revised: We note that this is also the space of meta-optimizers that learn to optimize (L2O) iterative update rules from data on a given task (Andrychowicz et al., 2016;

Original: Chen et al., 2017; Lv et al., 2017; Cao et al., 2019a).
Revised: Chen et al., 2017; Lv et al., 2017; Cao et al., 2019a).

Original: However, there has been no notion of uncertainty let alone the task of UQ for the learned optimizer in these L2O methods, which is to be addressed in our Bayesian L2O (BL2O).
Revised: However, there has been no notion of uncertainty let alone the task of UQ for the learned optimizer in these L2O methods, which is to be addressed in our Bayesian L2O (BL2O).

Original: 3. M ODELING AN OPTIMIZER AS A RANDOM VECTOR
Revised: 3. M ODELING AN OPTIMIZER AS A RANDOM VECTOR

Original: Now that we have the optimizer space G properly deﬁned and parameterized, we proceed to model an optimizer g as a random vector in the space.
Revised: Now that we have the optimizer space G properly deﬁned and parameterized, we proceed to model an optimizer g as a random vector in the space.

Original: Boltzmann-shaped Posterior.
Revised: Boltzmann-shaped Posterior.

Original: Since we have G = Θ , we can rewrite each g ∈ G as g θ with θ ∈ Θ and the optimal optimizer g ∗ as g θ ∗ .
Revised: Since we have G = Θ , we can rewrite each g ∈ G as g θ with θ ∈ Θ and the optimal optimizer g ∗ as g θ ∗ .

Original: Therefore, p ( g ∗ |D ) becomes p ( θ ∗ |D ) .
Revised: Therefore, p ( g ∗ |D ) becomes p ( θ ∗ |D ) .

Original: We consider a Gaussian prior over the parameters of the neural network : p ( θ ∗ ) ∝ exp( − λ ||
Revised: We consider a Gaussian prior over the parameters of the neural network : p ( θ ∗ ) ∝ exp( − λ ||

Original: θ ∗ || 22 ) , where λ is a constant controlling the variance.
Revised: θ ∗ || 22 ) , where λ is a constant controlling the variance.

Original: We use the chain rule to decompose the likelihood function p ( D| θ ∗ ) at a ﬁxed budge T :
Revised: We use the chain rule to decompose the likelihood function p ( D| θ ∗ ) at a ﬁxed budge T :

Original: [Equation]
Revised: [Equation]

Original: The second equality is due to that w t θ ∗ is ﬁxed given θ ∗ and past data points.
Revised: The second equality is due to that w t θ ∗ is ﬁxed given θ ∗ and past data points.

Original: For the single sample likelihood, we apply the results from Ortega et al
Revised: For the single sample likelihood, we apply the results from Ortega et al

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Cao & Shen
Revised: Cao & Shen

Original: [Equation]
Revised: [Equation]

Original: and obtain
Revised: and obtain

Original: [Equation]
Revised: [Equation]

Original: We multiply the likelihood functions of all samples together and obtain the Boltzmann-shaped likelihood function as p ( D| θ ∗ ) ∝ exp( − (cid:80) Tt =1 f ( w t θ ∗ )) .
Revised: We multiply the likelihood functions of all samples together and obtain the Boltzmann-shaped likelihood function as p ( D| θ ∗ ) ∝ exp( − (cid:80) Tt =1 f ( w t θ ∗ )) .

Original: We ﬁnally multiply the conjugate prior to the likelihood and obtain the Boltzmann-shaped posterior as:
Revised: We ﬁnally multiply the conjugate prior to the likelihood and obtain the Boltzmann-shaped posterior as:

Original: [Equation]
Revised: [Equation]

Original: where F ( θ ∗ ) =
Revised: where F ( θ ∗ ) =

Original: (cid:80) Tt =1 f ( w t θ ∗ )
Revised: (cid:80) Tt =1 f ( w t θ ∗ )

Original: + λ
Revised: + λ

Original: || θ
Revised: || θ

Original: ∗
Revised: ∗

Original: || 22 , which actually contains the objective in Eq 3 plus a L regularization constant.
Revised: || 22 , which actually contains the objective in Eq 3 plus a L regularization constant.

Original: Local Approximation and Bayesian Loss.
Revised: Local Approximation and Bayesian Loss.

Original: However, the above posterior distribution involves an integral in the normalization constant which is computationally intractable.
Revised: However, the above posterior distribution involves an integral in the normalization constant which is computationally intractable.

Original: Moreover, the architecture of F ( θ ∗ ) is so complicated that it is impossible to directly sample from the posterior distribution.
Revised: Moreover, the architecture of F ( θ ∗ ) is so complicated that it is impossible to directly sample from the posterior distribution.

Original: In order to overcome the aforementioned challenges, we would like to learn a distribution function q ( θ ∗ | φ ) that has the analytic form and is easy to be sampled, where φ is the parameter vector in q ( θ ∗ | φ ) , to approximate the real posterior p ( θ ∗ |D ) .
Revised: In order to overcome the aforementioned challenges, we would like to learn a distribution function q ( θ ∗ | φ ) that has the analytic form and is easy to be sampled, where φ is the parameter vector in q ( θ ∗ | φ ) , to approximate the real posterior p ( θ ∗ |D ) .

Original: Furthermore, due to the high dimensions of θ ∗ and the complicated landscape of the posterior, it is impossible to approximate p ( θ ∗ |D ) at every position in the θ ∗ space.
Revised: Furthermore, due to the high dimensions of θ ∗ and the complicated landscape of the posterior, it is impossible to approximate p ( θ ∗ |D ) at every position in the θ ∗ space.

Original: We then consider to approximate it locally around θ c , an optimum of interest for F ( θ ∗ ) .
Revised: We then consider to approximate it locally around θ c , an optimum of interest for F ( θ ∗ ) .

Original: We denote the local region as Θ c , a neighborhood around θ c , and re-normalization constant C = (cid:82) θ
Revised: We denote the local region as Θ c , a neighborhood around θ c , and re-normalization constant C = (cid:82) θ

Original: [Equation]
Revised: [Equation]

Original: Θ c p ( θ ∗ |D )
Revised: Θ c p ( θ ∗ |D )

Original: d θ ∗ .
Revised: d θ ∗ .

Original: Then the local posterior will be a conditioned (re-scaled) version of p ( θ ∗ |D ) : p
Revised: Then the local posterior will be a conditioned (re-scaled) version of p ( θ ∗ |D ) : p

Original: [Equation]
Revised: [Equation]

Original: ( θ
Revised: ( θ

Original: [Equation]
Revised: [Equation]

Original: |D )
Revised: |D )

Original: =
Revised: =

Original: p
Revised: p

Original: [Equation]
Revised: [Equation]

Original: θ ∗ |D ) /C, θ ∗ ∈ Θ c .
Revised: θ ∗ |D ) /C, θ ∗ ∈ Θ c .

Original: In order to make q ( θ ∗ | φ ) ≈ p (cid:48) ( θ ∗ |D ) , we calculate the
Revised: In order to make q ( θ ∗ | φ ) ≈ p (cid:48) ( θ ∗ |D ) , we calculate the

Original: KL-divergence between these two:
Revised: KL-divergence between these two:

Original: [Equation]
Revised: [Equation]

Original: where Z = (cid:82) exp( − F ( θ ∗ ))
Revised: where Z = (cid:82) exp( − F ( θ ∗ ))

Original: d θ ∗ is the normalization constant.
Revised: d θ ∗ is the normalization constant.

Original: The second term in the above equation equals to log( ZC ) , a constant w.r.t.
Revised: The second term in the above equation equals to log( ZC ) , a constant w.r.t.

Original: φ , thus could be ignored during optimization.
Revised: φ , thus could be ignored during optimization.

Original: We then propose our Bayesian loss as:
Revised: We then propose our Bayesian loss as:

Original: [Equation]
Revised: [Equation]

Original: where the ﬁrst term of F B measures the negative entropy of our approximated posterior, and the second term is the expectation of the loss function over of posterior.
Revised: where the ﬁrst term of F B measures the negative entropy of our approximated posterior, and the second term is the expectation of the loss function over of posterior.

Original: Gaussian Posterior.
Revised: Gaussian Posterior.

Original: For local approximation, we consider φ = ( µ , Σ ) and q ( θ ∗ | φ ) =
Revised: For local approximation, we consider φ = ( µ , Σ ) and q ( θ ∗ | φ ) =

Original: N ( µ , Σ ) , where µ is the mean vector and Σ is the covariance matrix of a normal distribution.
Revised: N ( µ , Σ ) , where µ is the mean vector and Σ is the covariance matrix of a normal distribution.

Original: For simplicity, we consider Σ to be a diagonal matrix: Σ = diag ( σ 21 , σ 22 , σ 23 , ... ) .
Revised: For simplicity, we consider Σ to be a diagonal matrix: Σ = diag ( σ 21 , σ 22 , σ 23 , ... ) .

Original: The second term in Eq (9) involves the integral over F ( θ ∗ ) , which is intractable.
Revised: The second term in Eq (9) involves the integral over F ( θ ∗ ) , which is intractable.

Original: Therefore, we use Monte Carlo sampling through q ( θ ∗ | φ ) to replace the integral there.
Revised: Therefore, we use Monte Carlo sampling through q ( θ ∗ | φ ) to replace the integral there.

Original: However, the direct sampling of the posterior parameters makes it difﬁcult for the optimization as it is inaccessible to get the gradient w.r.t.
Revised: However, the direct sampling of the posterior parameters makes it difﬁcult for the optimization as it is inaccessible to get the gradient w.r.t.

Original: µ and Σ .
Revised: µ and Σ .

Original: Moreover, the standard deviation σ 1 , σ 2 , ... must be non-negative, making the optimization constrained.
Revised: Moreover, the standard deviation σ 1 , σ 2 , ... must be non-negative, making the optimization constrained.

Original: To overcome those two challenges, we use the trick introduced in (Blundell et al., 2015) to shift sampling from q ( θ ∗ | φ ) to sampling from a standard normal distribution N ( 0 , I ) .
Revised: To overcome those two challenges, we use the trick introduced in (Blundell et al., 2015) to shift sampling from q ( θ ∗ | φ ) to sampling from a standard normal distribution N ( 0 , I ) .

Original: And we reparameterize standard deviation σ i to ρ i as σ i =
Revised: And we reparameterize standard deviation σ i to ρ i as σ i =

Original: log(1+exp( ρ i )) .
Revised: log(1+exp( ρ i )) .

Original: Then for any (cid:15) sampled from N ( µ , I ) , we could calculate θ ∗ as θ ∗ =
Revised: Then for any (cid:15) sampled from N ( µ , I ) , we could calculate θ ∗ as θ ∗ =

Original: u + log(1 + exp( ρ )) (cid:12) (cid:15) , where (cid:12) means element-wise product and ρ = ( ρ 1 , ρ 2 , ... ) .
Revised: u + log(1 + exp( ρ )) (cid:12) (cid:15) , where (cid:12) means element-wise product and ρ = ( ρ 1 , ρ 2 , ... ) .

Original: 3. B AYESIAN A VERAGING
Revised: 3. B AYESIAN A VERAGING

Original: We recall our goal to build the posterior over the global optimum: p ( w ∗ |D ) through Eq 4.
Revised: We recall our goal to build the posterior over the global optimum: p ( w ∗ |D ) through Eq 4.

Original: We consider using Monte Carlo sampling to approximate the integral as:
Revised: We consider using Monte Carlo sampling to approximate the integral as:

Original: [Equation]
Revised: [Equation]

Original: where θ ∗ i is sampled from q ( θ ∗ | φ ) .
Revised: where θ ∗ i is sampled from q ( θ ∗ | φ ) .

Original: Since q ( θ ∗ | φ ) follows a multivariate Gaussian distribution where individual dimensions are independent of each other, we estimate the summation above in each dimension using independent MC samplings.
Revised: Since q ( θ ∗ | φ ) follows a multivariate Gaussian distribution where individual dimensions are independent of each other, we estimate the summation above in each dimension using independent MC samplings.

Original: In practice, N being 10 , 000 , 100 , 000 , and 500 , 000 led to negligible differences in the 1D estimations, and N was thus ﬁxed at 10 , 000 .
Revised: In practice, N being 10 , 000 , 100 , 000 , and 500 , 000 led to negligible differences in the 1D estimations, and N was thus ﬁxed at 10 , 000 .

Original: 3. M ETA - TRAINING S ET
Revised: 3. M ETA - TRAINING S ET

Original: In order to boost the robustness and generalizability of our optimizer posterior p ( g ∗ |D ) , we consider using an ensemble of objective functions F = { f i } Ni =1 .
Revised: In order to boost the robustness and generalizability of our optimizer posterior p ( g ∗ |D ) , we consider using an ensemble of objective functions F = { f i } Ni =1 .

Original: Speciﬁcally, we replace the objective function in Eq 3 with 1 N (cid:80) Ni =1 (cid:80)
Revised: Speciﬁcally, we replace the objective function in Eq 3 with 1 N (cid:80) Ni =1 (cid:80)

Original: Tt =
Revised: Tt =

Original: 1 f i ( w t θ ∗ ,i ) and rewrite F ( θ ∗ ) as:
Revised: 1 f i ( w t θ ∗ ,i ) and rewrite F ( θ ∗ ) as:

Original: [Equation]
Revised: [Equation]

Original: (11) where w t θ ∗ ,i is the solution at t th iteration for objective f i optimized by g θ ∗ .
Revised: where w t θ ∗ ,i is the solution at t th iteration for objective f i optimized by g θ ∗ .

Original: Such replacement can let our posterior generalize to novel objective functions.
Revised: Such replacement can let our posterior generalize to novel objective functions.

Original: We regard the functional dataset F as the meta-training set.
Revised: We regard the functional dataset F as the meta-training set.

Original: During the experiments, we create different meta-training sets for different problems, which will be described in details in Sec 4.
Revised: During the experiments, we create different meta-training sets for different problems, which will be described in details in Sec 4.

Original: We note that Eq 11 is also the objective or part of the objective in many meta-optimizers (Andrychowicz et al., 2016;
Revised: We note that Eq 11 is also the objective or part of the objective in many meta-optimizers (Andrychowicz et al., 2016;

Original: Chen et al., 2017; Lv et al., 2017; Cao et al., 2019a).
Revised: Chen et al., 2017; Lv et al., 2017; Cao et al., 2019a).

Original: However, those methods are focusing on training a deterministic optimizer without uncertainty-awareness.
Revised: However, those methods are focusing on training a deterministic optimizer without uncertainty-awareness.

Original: 3. D EFINING THE L OCAL R EGION THROUGH P RE - TRAINING  .
Revised: 3. T WO TRAINING STAGES FOE EMPOWERING THE LOCAL POSTERIOR .

Original: Aswe mentioned before, due to the extreme large optimizer space, we focus on modelling the posterior locally around θ c , an optimum of interest.
Revised: As mentioned before, due to the extreme large optimizer space, we focus on modelling the posterior locally around θ c , an optimum of interest.

Revised: If we directly train our model through the Bayesian loss in Eq 9, we simply regard that our posterior is locally around the random initialized point.

Original: To obtain  θ c , we ﬁrst pre-train our model in a non-Bayesian way through minimizing the loss in Eq 11.
Revised: In order to obtain an real optimum of interest θ c , we ﬁrst train our model in a non-Bayesian way through minimizing the loss in Eq 11.

Original: We then use θ c as the warm start for µ , and start the second Bayesian training stage.
Revised: We then use θ c as the warm start for µ , and start the second Bayesian training stage through the loss in Eq 9.

Revised: Both training stages are critical for empowering our local posterior.

Original: We demonstrate the importance of pre-training through the ablation study in Appendix F.
Revised: Such statement can be demonstrated through the ablation study in Appendix F.

Original: 3. M ODEL A RCHITECTURE , I MPLEMENTATION AND C OMPUTATIONAL C OMPLEXITY
Revised: 3. M ODEL A RCHITECTURE , I MPLEMENTATION AND C OMPUTATIONAL C OMPLEXITY

Original: The model is implemented in Tensorﬂow 1.13 (Abadi et al., 2016) and optimized by Adam (Kingma & Ba, 2014).
Revised: The model is implemented in Tensorﬂow 1.13 (Abadi et al., 2016) and optimized by Adam (Kingma & Ba, 2014).

Original: For the optimizer architecture, we use the coordinate-wise LSTM from Andrychowicz et al.
Revised: For the optimizer architecture, we use the coordinate-wise LSTM from Andrychowicz et al.

Original: We also validate this design choice in Appendix F.
Revised: We also validate this design choice in Appendix F.

Original: Due to the coordinate-wise nature, our BL2O model only contains 10,282 free parameters.
Revised: Due to the coordinate-wise nature, our BL2O model only contains 10,282 free parameters.

Original: For all experiments, the length of LSTM is set to be 20.
Revised: For all experiments, the length of LSTM is set to be 20.

Original: Both training stages include 5,000 training epochs.
Revised: Both training stages include 5,000 training epochs.

Original: The time complexity for BL2O is O ( KBN e + KN e H 2 ) , where K is the number of sampling trajectories, B is the minibatch size, N e is the number of objective parameters, and H is the hidden size of LSTM ( H = 20 in the study).
Revised: The time complexity for BL2O is O ( KBN e + KN e H 2 ) , where K is the number of sampling trajectories, B is the minibatch size, N e is the number of objective parameters, and H is the hidden size of LSTM ( H = 20 in the study).

Original: As the batch size increases, the computational cost is close to the traditional Bayesian neural networks trained through SGD.
Revised: As the batch size increases, the computational cost is close to the traditional Bayesian neural networks trained through SGD.

Original: Due to the coordinate-wise LSTM, the space coomplexity (memory cost) of BL2O is only O
Revised: Due to the coordinate-wise LSTM, the space coomplexity (memory cost) of BL2O is only O

Original: ( H 2 ) , which remains the same as the number of objective parameters varies.
Revised: ( H 2 ) , which remains the same as the number of objective parameters varies.

Original: Both the time and the space complexity of BL2O are the same as
Revised: Both the time and the space complexity of BL2O are the same as

Original: DM LSTM (Abadi et al., 2016), while those of Adam are O ( KBN e ) and O ( N e ) , respectively.
Revised: DM LSTM (Abadi et al., 2016), while those of Adam are O ( KBN e ) and O ( N e ) , respectively.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: We test our BL2O model extensively on optimizing: non-convex test functions, energy functions in protein-protein interactions, loss functions in image classiﬁcation and loss functions in data privacy attack.
Revised: We test our BL2O model extensively on optimizing: non-convex test functions, energy functions in protein-protein interactions, loss functions in image classiﬁcation and loss functions in data privacy attack.

Original: We compare BL2O to three non-Bayesian methods: Adam, Particle Swarm Optimization (PSO) (Kennedy & Eberhart, 1995), DM LSTM (Andrychowicz et al., 2016) and a recently published Bayesian method, BAL (Cao & Shen, 2020).
Revised: We compare BL2O to three non-Bayesian methods: Adam, Particle Swarm Optimization (PSO) (Kennedy & Eberhart, 1995), DM LSTM (Andrychowicz et al., 2016) and a recently published Bayesian method, BAL (Cao & Shen, 2020).

Original: All algorithms are running for 10,000 times with random initializing points to obtain the empirical posterior distributions.
Revised: All algorithms are running for 10,000 times with random initializing points to obtain the empirical posterior distributions.

Original: During each run, the hyperparameters in Adam and PSO are sampled from Table 4 in Appendix A.
Revised: During each run, the hyperparameters in Adam and PSO are sampled from Table 4 in Appendix A.

Original: Out of 10,000 solutions we choose the one with the lowest function value to be the ﬁnal solution ( ˆ w ).
Revised: Out of 10,000 solutions we choose the one with the lowest function value to be the ﬁnal solution ( ˆ w ).

Original: Generally, for optimization performance, we assess the distance between the ﬁnal solution and the global optima:
Revised: Generally, for optimization performance, we assess the distance between the ﬁnal solution and the global optima:

Original: ||
Revised: ||

Original: ˆ w − w ∗
Revised: ˆ w − w ∗

Original: || .
Revised: || .

Original: The lower the distance is, the better the solution quality is.
Revised: The lower the distance is, the better the solution quality is.

Original: For uncertainty quantiﬁcation, we assess the upper bound r σ and the real conﬁdence (cid:15) σ given a ﬁxed conﬁdence level σ .
Revised: For uncertainty quantiﬁcation, we assess the upper bound r σ and the real conﬁdence (cid:15) σ given a ﬁxed conﬁdence level σ .

Original: The real conﬁdence (cid:15) σ is deﬁned by the fraction of 10,000 solutions that actually fall in the bounded region.
Revised: The real conﬁdence (cid:15) σ is deﬁned by the fraction of 10,000 solutions that actually fall in the bounded region.

Original: The lower the r σ is, the tighter the conﬁdence interval is.
Revised: The lower the r σ is, the tighter the conﬁdence interval is.

Original: And the closer of (cid:15) σ to σ is, the more accurate the conﬁdence estimate is.
Revised: And the closer of (cid:15) σ to σ is, the more accurate the conﬁdence estimate is.

Original: Comparison in optimizing test functions.
Revised: Comparison in optimizing test functions.

Original: We ﬁrst test the performance on test functions in the global optimization benchmark set (Jamil & Yang, 2013).
Revised: We ﬁrst test the performance on test functions in the global optimization benchmark set (Jamil & Yang, 2013).

Original: We choose three extremely rugged, nonconvex functions: Rastrigin, Ackley and Griewank in 5 dimensions: 6D, 12D, 18D, 24D, 30D. For each function, we create a diverse, broad family of similar functions f j ( w ) as the meta-training set used for training DM LSTM and BL2O. The analytical forms and the meta-training sets of those functions are shown in Table 5 in Appendix B.
Revised: We choose three extremely rugged, nonconvex functions: Rastrigin, Ackley and Griewank in 5 dimensions: 6D, 12D, 18D, 24D, 30D. For each function, we create a diverse, broad family of similar functions f j ( w ) as the meta-training set used for training DM LSTM and BL2O. The analytical forms and the meta-training sets of those functions are shown in Table 5 in Appendix B.

Original: We compare BL2O with all 4 competing methods.
Revised: We compare BL2O with all 4 competing methods.

Original: The optimization and UQ performances are shown in Fig.
Revised: The optimization and UQ performances are shown in Fig.

Original: In all three cases and 5 dimensions, BL2O has led to the best solution quality.
Revised: In all three cases and 5 dimensions, BL2O has led to the best solution quality.

Original: In terms of UQ, BL2O has shown the most accurate conﬁdence estimation ( (cid:15) σ ≈ σ ) when σ = 0 .
Revised: In terms of UQ, BL2O has shown the most accurate conﬁdence estimation ( (cid:15) σ ≈ σ ) when σ = 0 .

Original: 9 and σ = 0 .
Revised: 9 and σ = 0 .

Original: 8 , while BAL was the second best.
Revised: 8 , while BAL was the second best.

Original: And BL2O has shown much tighter conﬁdence intervals r σ against BAL.
Revised: And BL2O has shown much tighter conﬁdence intervals r σ against BAL.

Original: In some cases, although DM LSTM has lower r σ than BL2O, it has much lower conﬁdence level, indicating that this tight upper bound in DM LSTM is miscalibrated.
Revised: In some cases, although DM LSTM has lower r σ than BL2O, it has much lower conﬁdence level, indicating that this tight upper bound in DM LSTM is miscalibrated.

Original: As a result,
Revised: As a result,

Original: BL2O has shown the best performance in both optimization and UQ.
Revised: BL2O has shown the best performance in both optimization and UQ.

Original: [Figure]
Revised: [Figure]

Original: Comparison in optimizing energy functions for protein docking.
Revised: Comparison in optimizing energy functions for protein docking.

Original: We then apply BL2O to a bioinformatics application: predicting the 3D structures of protein-complexes (Smith & Sternberg, 2002), called protein docking.
Revised: We then apply BL2O to a bioinformatics application: predicting the 3D structures of protein-complexes (Smith & Sternberg, 2002), called protein docking.

Original: Ab initio protein docking can be recast as optimizing a noisy and expensive energy function in a high-dimensional conformational space (Cao & Shen, 2020): x ∗ = arg min x f ( x ) .
Revised: Ab initio protein docking can be recast as optimizing a noisy and expensive energy function in a high-dimensional conformational space (Cao & Shen, 2020): x ∗ = arg min x f ( x ) .

Original: While solving such optimization problems still remains difﬁcult, quantifying the uncertainty of resulting optima (docking solutions) is even more challenging.
Revised: While solving such optimization problems still remains difﬁcult, quantifying the uncertainty of resulting optima (docking solutions) is even more challenging.

Original: In this section, we apply our BL2O to optimization and uncertainty quantiﬁcation in protein docking and compare with a state-of-the-art method BAL (Cao & Shen, 2020).
Revised: In this section [Figure] we apply our BL2O to optimization and uncertainty quantiﬁcation in protein docking and compare with a state-of-the-art method BAL (Cao & Shen, 2020).

Original: We describe the detailed settings of BL2O on protein docking in Appendix C. From BL2O, we obtain a posterior distribution p ( w ∗ | D ) over the native structure w ∗ and the lowest energy structure, ˆ w .
Revised: We describe the detailed settings of BL2O on protein docking in Appendix C. From BL2O, we obtain a posterior distribution p ( w ∗ | D ) over the native structure w ∗ and the lowest energy structure, ˆ w .

Original: In protein docking, the quality of a predicted structure is based on the distance to the native structure (the global optimum): || ˆ w − w ∗
Revised: In protein docking, the quality of a predicted structure is based on the distance to the native structure (the global optimum): || ˆ w − w ∗

Original: || .
Revised: || .

Original: For UQ, we assess the two-sided conﬁdence interval at σ = 0 .as
Revised: For UQ, we assess the two-sided conﬁdence interval at σ = 0 .as

Original: P ( l 0 .
Revised: P ( l 0 .

Original: 9 (cid:54) ||
Revised: 9 (cid:54) ||

Original: ˆ w − w ∗
Revised: ˆ w − w ∗

Original: ||
Revised: ||

Original: (cid:54) r 0 . 9 )
Revised: (cid:54) r 0 . 9 )

Original: = 0 .
Revised: = 0 .

Original: In Table 1, we assess || ˆ w − w ∗
Revised: In Table [Table] we assess || ˆ w − w ∗

Original: || , r 0 .
Revised: || , r 0 .

Original: 9 − l 0 .
Revised: 9 − l 0 .

Original: 9 and whether || ˆ w − w ∗
Revised: 9 and whether || ˆ w − w ∗

Original: || is within the conﬁdence interval.
Revised: || is within the conﬁdence interval.

Original: For optimization, BL2O clearly outperforms BAL in two medium cases while performing slightly worse in the other cases.
Revised: For optimization, BL2O clearly outperforms BAL in two medium cases while performing slightly worse in the other cases.

Original: Yet for UQ, BL2O shows clearly superior performance over BAL in all cases, with accurate or/and tight conﬁdence intervals.
Revised: Yet for UQ, BL2O shows clearly superior performance over BAL in all cases, with accurate or/and tight conﬁdence intervals.

Original: We also visulize the posterior distributions over || ˆ w − w ∗ || for protein 1JMO 4.
Revised: We also visulize the posterior distributions over || ˆ w − w ∗ || for protein 1JMO 4.

Original: As shown in Fig 2, we can see compared to that of BAL,
Revised: As shown in Fig 2, we can see compared to that of BAL,

Original: BL2O’s distribution has real ||
Revised: BL2O’s distribution has real ||

Original: ˆ w − w ∗ || within the 90% C.I. and smaller variance.
Revised: ˆ w − w ∗ || within the 90% C.I. and smaller variance.

Original: More posterior distributions are shown in Appendix D.
Revised: More posterior distributions are shown in Appendix D.

Original: [Table]
Revised: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]
Revised: [Figure]

Original: Comparison in optimizing loss functions in image classiﬁcation.
Revised: Comparison in optimizing loss functions in image classiﬁcation.

Original: We then test the performance of optimizing the loss function in image classiﬁcation on the MNIST dataset.
Revised: We then test the performance of optimizing the loss function in image classiﬁcation on the MNIST dataset.

Original: We apply a 2layers MLP network as the classiﬁer.
Revised: We apply a 2layers MLP network as the classiﬁer.

Original: The competing methods include Adam, DM LSTM and two
Revised: The competing methods include Adam, DM LSTM and two

Original: Bayesian neural network methods: variational inference (VI) (Blundell et al., 2015) and Learnable
Revised: Bayesian neural network methods: variational inference (VI) (Blundell et al., 2015) and Learnable

Original: Bernoulli Dropout (LBO) (Boluki et al., 2020).
Revised: Bernoulli Dropout (LBO) (Boluki et al., 2020).

Original: Moreover, for DM LSTM and BL2O, we apply a trick during the optimizer training called curriculum learning (CL) and introduce it in detail in Appendix E for training over long-term iterations.
Revised: Moreover, for DM LSTM and BL2O, we apply a trick during the optimizer training called curriculum learning (CL) and introduce it in detail in Appendix E for training over long-term iterations.

Original: We call DM LSTM with CL as DM LSTM C and
Revised: We call DM LSTM with CL as DM LSTM C and

Original: BL2O with CL as BL2O C.
Revised: BL2O with CL as BL2O C.

Original: The assessment of the optimization and UQ for this machine learning task is different from that for optimization before.
Revised: The assessment of the optimization and UQ for this machine learning task is different from that for optimization before.

Original: In terms of optimization, we assess the classiﬁcation accuracy on the test set.
Revised: In terms of optimization, we assess the classiﬁcation accuracy on the test set.

Original: In terms of UQ, we measure two metrics that assess the robustness and trustworthiness of the classiﬁer: the in-domain calibration error and the out-of-domain detection rate.
Revised: In terms of UQ, we measure two metrics that assess the robustness and trustworthiness of the classiﬁer: the in-domain calibration error and the out-of-domain detection rate.

Original: We ﬁrst compare the accuracy on the testing set among different methods.
Revised: We ﬁrst compare the accuracy on the testing set among different methods.

Original: As shown in Table 2,
Revised: As shown in Table 2,

Original: Adam, DM LSTM C and BL2O C have almost the same best performance.
Revised: Adam, DM LSTM C and BL2O C have almost the same best performance.

Original: The signiﬁcant improvement from DM LSTM to DM LSTM C, and from BL2O to BL2O C shows the big advantage of curriculum learning in learning to optimize.
Revised: The signiﬁcant improvement from DM LSTM to DM LSTM C, and from BL2O to BL2O C shows the big advantage of curriculum learning in learning to optimize.

Original: In conclusion, BL2O C had on par accuracy with
Revised: In conclusion, BL2O C had on par accuracy with

Original: Adam and DM LSTM C on the MNIST dataset.
Revised: Adam and DM LSTM C on the MNIST dataset.

Original: However, classiﬁcation models must not only be accurate, but also indicate when they are likely to be incorrect.
Revised: However, classiﬁcation models must not only be accurate, but also indicate when they are likely to be incorrect.

Original: Conﬁdence calibration, the probability that estimates the true likelihood of each prediction is also important for classiﬁcation models.
Revised: Conﬁdence calibration, the probability that estimates the true likelihood of each prediction is also important for classiﬁcation models.

Original: In the ideal case, the maximum output probability (MaxConﬁdence) for each test sample should be equal to the prediction accuracy for that sample.
Revised: In the ideal case, the maximum output probability (MaxConﬁdence) for each test sample should be equal to the prediction accuracy for that sample.

Original: To assess the calibration of each methods, we split the test set into 20 equal-sized bins and assess the calibration error as the average discrepancy between accuracy and MaxConﬁdence in each bin.
Revised: To assess the calibration of each methods, we split the test set into 20 equal-sized bins and assess the calibration error as the average discrepancy between accuracy and MaxConﬁdence in each bin.

Original: As seen in Table 2, among all methods compared, BL2O C and BL2O had the least calibration error.
Revised: As seen in Table 2, among all methods compared, BL2O C and BL2O had the least calibration error.

Original: The ﬁgure of Acc. vs MaxConf. is also shown in Fig.
Revised: The ﬁgure of Acc. vs MaxConf. is also shown in Fig.

Original: 4 in Appendix E.
Revised: 4 in Appendix E.

Original: We also inspect the out-of-domain detection of BL2O, BL2O C and competing methods.
Revised: We also inspect the out-of-domain detection of BL2O, BL2O C and competing methods.

Original: We train all models on the data belonging to the ﬁrst 5 classes in the MNIST training dataset (the last layer of the optimizee is modiﬁed to have 5 rather than 10 neurons) and test them on the remainingsamples from the other 5 classes. An ideal model would predict a uniform distribution over the
Revised: We train all models on the data belonging to the ﬁrst 5 classes in the MNIST training dataset (the last layer of the optimizee is modiﬁed to have 5 rather than 10 neurons) and test them on the remaining

Original: [Table]
Revised: [Table]

Original: wrong classes.
Revised: samples from the other 5 classes. An ideal model would predict a uniform distribution over thewrong classes.

Original: Therefore, we deﬁne the out-of-domain detection rate at threshold t , q t , as the percentage of test samples with max class conﬁdence below t .
Revised: Therefore, we deﬁne the out-of-domain detection rate at threshold t , q t , as the percentage of test samples with max class conﬁdence below t .

Original: The larger the q t , the better out-ofdomain detection is.
Revised: The larger the q t , the better out-ofdomain detection is.

Original: As shown in Table 2, BL2O and BL2O C shows superior performance with all competing methods.
Revised: As shown in Table 2, BL2O and BL2O C shows superior performance with all competing methods.

Original: Notably, BL2O without curriculum learning had much better out-of-domain detection rates compared to BL2O with curriculum learning.
Revised: Notably, BL2O without curriculum learning had much better out-of-domain detection rates compared to BL2O with curriculum learning.

Original: Comparison in optimizing loss functions for data privacy attack.
Revised: Comparison in optimizing loss functions for data privacy attack.

Original: We ﬁnally apply our model to an application that critically needs UQ.
Revised: We ﬁnally apply our model to an application that critically needs UQ.

Original: As many machine learning models are deployed publicly, it is important to avoid leaking private sensitive information, such as ﬁnancial data, health data and so on.
Revised: As many machine learning models are deployed publicly, it is important to avoid leaking private sensitive information, such as ﬁnancial data, health data and so on.

Original: Data privacy attack (Nasr et al., 2018) studies this problem by playing the role of hackers and attacking the machine-learning models to quantify the risk of privacy leakage.
Revised: Data privacy attack (Nasr et al., 2018) studies this problem by playing the role of hackers and attacking the machine-learning models to quantify the risk of privacy leakage.

Original: Better attacks would help models to be better prepared for privacy defense.
Revised: Better attacks would help models to be better prepared for privacy defense.

Original: We use the model and dataset in (Cao et al., 2019b), where each input has 9 features involving patient genetic information and the output p is the probability of the clinical signiﬁcance (having cancer or not) for a patient.
Revised: We use the model and dataset in (Cao et al., 2019b), where each input has 9 features involving patient genetic information and the output p is the probability of the clinical signiﬁcance (having cancer or not) for a patient.

Original: We study the following model inversion attack (Fredrikson et al., 2015): by giving 5 features w (cid:48) ∈ [0 , 1] 5 out of 9 and the label p of each patient, we want to recover the rest 4 features w ∗ ∈ [0 , 1] 4 (potentially sensitive patient information).
Revised: We study the following model inversion attack (Fredrikson et al., 2015): by giving 5 features w (cid:48) ∈ [0 , 1] 5 out of 9 and the label p of each patient, we want to recover the rest 4 features w ∗ ∈ [0 , 1] 4 (potentially sensitive patient information).

Original: Therefore, for each patient, the objective is w ∗ =
Revised: Therefore, for each patient, the objective is w ∗ =

Original: arg min w ∈ [0 , 1] 4 ( m ( w (cid:48) , w ) − p ) 2 , where w ∗ is the ground-truth of w and m is the trained predictive model.
Revised: arg min w ∈ [0 , 1] 4 ( m ( w (cid:48) , w ) − p ) 2 , where w ∗ is the ground-truth of w and m is the trained predictive model.

Original: The closeness between the predicted and the real input features can quantify the risk of information leakage and the quality of the attack.
Revised: The closeness between the predicted and the real input features can quantify the risk of information leakage and the quality of the attack.

Original: We compare BL2O with
Revised: We compare BL2O with

Original: Adam, PSO, BAL and DM LSTM on optimization and UQ on all test cases in (Cao et al., 2019b).
Revised: Adam, PSO, BAL and DM LSTM on optimization and UQ on all test cases in (Cao et al., 2019b).

Original: The meta-training objectives for BL2O and DM LSTM are the training set in (Cao et al., 2019b).
Revised: The meta-training objectives for BL2O and DM LSTM are the training set in (Cao et al., 2019b).

Original: As shown in Table 3, BL2O has shown the best performance in both optimization and UQ compared to all competing methods.
Revised: As shown in Table 3, BL2O has shown the best performance in both optimization and UQ compared to all competing methods.

Original: It is noteworthy that learned optimizers (DM LSTM and BL2O) had much better optimization performance than pre-deﬁned optimizers.
Revised: It is noteworthy that learned optimizers (DM LSTM and BL2O) had much better optimization performance than pre-deﬁned optimizers.

Original: And the Bayesian methods (BAL and BL2O) had signiﬁcantly better UQ performance than non-Bayesian methods.
Revised: And the Bayesian methods (BAL and BL2O) had signiﬁcantly better UQ performance than non-Bayesian methods.

Original: BL2O possessed the advantages of both learned and Bayesian optimizers to achieve the best performance.
Revised: BL2O possessed the advantages of both learned and Bayesian optimizers to achieve the best performance.

Original: [Table]
Revised: [Table]

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: Current optimization algorithms, even with uncertainty-awareness, do not address the uncertainty arising within the optimizer itself.
Revised: Current optimization algorithms, even with uncertainty-awareness, do not address the uncertainty arising within the optimizer itself.

Original: To close this gap, we parameterize the update rule as a neural network and build a Boltzmann-shaped posterior over the algorithmic space.
Revised: To close this gap, we parameterize the update rule as a neural network and build a Boltzmann-shaped posterior over the algorithmic space.

Original: We apply our Bayesian
Revised: We apply our Bayesian

Original: Learning-to-Optimize (BL2O) framework to optimize test functions, energy functions in protein docking, loss functions in image classiﬁcation and loss functions in data privacy attack.
Revised: Learning-to-Optimize (BL2O) framework to optimize test functions, energy functions in protein docking, loss functions in image classiﬁcation and loss functions in data privacy attack.

Original: The empirical results demonstrate that BL2O outperforms the state-of-the-art methods in both optimization and uncertainty quantiﬁcation, as well as the calibration and out-of-domain detection in classiﬁcation.
Revised: The empirical results demonstrate that BL2O outperforms the state-of-the-art methods in both optimization and uncertainty quantiﬁcation, as well as the calibration and out-of-domain detection in classiﬁcation.

Original: A O PTIMIZER DISTRIBUTION SETTINGS FOR A DAM AND PSO
Revised: A O PTIMIZER DISTRIBUTION SETTINGS FOR A DAM AND PSO

Original: [Table]
Revised: [Table]

Original: B A NALYTIC FORMS AND M ETA - TRAINING SETS OF T EST FUNCTIONS
Revised: B A NALYTIC FORMS AND M ETA - TRAINING SETS OF T EST FUNCTIONS

Original: [Table] [Table]
Revised: [Table] [Table]

Original: C S ETTINGS FOR P ROTEIN D OCKING E XPERIMENTS
Revised: C S ETTINGS FOR P ROTEIN D OCKING E XPERIMENTS

Original: We calculate the energy function (objective function f ( x ) ) in a CHARMM 19 force ﬁeld as in (Moal & Bates, 2010).
Revised: We calculate the energy function (objective function f ( x ) ) in a CHARMM 19 force ﬁeld as in (Moal & Bates, 2010).

Original: 25 protein-protein complexes are chosen from the protein docking benchmark set 4.0 (Hwang et al., 2010) as the training set, which is shown in Table 6.
Revised: 25 protein-protein complexes are chosen from the protein docking benchmark set 4.0 (Hwang et al., 2010) as the training set, which is shown in Table 6.

Original: For each target, we choosestarting points (top-5 models from ZDOCK (Pierce et al., 2014)).
Revised: For each target, we choosestarting points (top-5 models from ZDOCK (Pierce et al., 2014)).

Original: In total, our training set includes 125 samples.
Revised: In total, our training set includes 125 samples.

Original: Moreover, we parameterize the search space as R 12 as in BAL (Cao & Shen, 2020).
Revised: Moreover, we parameterize the search space as R 12 as in BAL (Cao & Shen, 2020).

Original: The resulting f ( x ) is fully differentiable in the search space.
Revised: The resulting f ( x ) is fully differentiable in the search space.

Original: We only consider 100 interface atoms due to the computational concern.
Revised: We only consider 100 interface atoms due to the computational concern.

Original: The number of iterations for one training epoch is 600 and in total we have 5000 training epochs.
Revised: The number of iterations for one training epoch is 600 and in total we have 5000 training epochs.

Original: Both BL2O and BAL have 600 iterations during the testing stage.
Revised: Both BL2O and BAL have 600 iterations during the testing stage.

Original: For fair comparison, after optimization, we rescore the BL2O samples for UQ using the scoring function (random forest) in BAL.
Revised: For fair comparison, after optimization, we rescore the BL2O samples for UQ using the scoring function (random forest) in BAL.

Original: [Table]
Revised: [Table]

Original: D P OSTERIOR D ISTRIBUTIONS IN P ROTEIN D OCKING
Revised: D P OSTERIOR D ISTRIBUTIONS IN P ROTEIN D OCKING

Original: || w * , ̂ w || 0.
Revised: || w * , ̂ w || 0.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: D
Revised: D

Original: [Figure]
Revised: [Figure]

Original: n s
Revised: n s

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: ||
Revised: ||

Original: [Table]
Revised: [Table]

Original: , ̂
Revised: , ̂

Original: [Table]
Revised: [Table]

Original: ||
Revised: ||

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: || w * , ̂ w ||
Revised: || w * , ̂ w ||

Original: [Table]
Revised: [Table]

Original: 0̂ 0̂
Revised: 0̂ 0̂

Original: 1̂ 1̂
Revised: 1̂ 1̂

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: || w
Revised: || w

Original: [Table] [Table]
Revised: [Table] [Table]

Original: w ||
Revised: w ||

Original: [Table]
Revised: [Table]

Original: 1̂
Revised: 1̂

Original: [Table]
Revised: [Table]

Original: || w * , ̂ w || 0.
Revised: || w * , ̂ w || 0.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table]
Revised: [Table]

Original: || w
Revised: || w

Original: [Table]
Revised: [Table]

Original: ̂ w ||
Revised: ̂ w ||

Original: [Table] [Table]
Revised: [Table] [Table]

Original: ||w * , ̂w||
Revised: ||w * , ̂w||

Original: [Figure]
Revised: [Figure]

Original: 0̂ 0̂
Revised: 0̂ 0̂

Original: 1̂
Revised: 1̂

Original: 1̂ 1̂
Revised: 1̂ 1̂

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Figure]
Revised: [Figure]

Original: E C URRICULUM L EARNING AND C ALIBRATION F IGURE IN I MAGE CLASSIFICATION
Revised: E C URRICULUM L EARNING AND C ALIBRATION F IGURE IN I MAGE CLASSIFICATION

Original: A common issue in learning to optimize for neural network training is that: the optimizer training usually takes hundreds of iterations, while training a neural network usually costs thousands or tens of thousand iterations.
Revised: A common issue in learning to optimize for neural network training is that: the optimizer training usually takes hundreds of iterations, while training a neural network usually costs thousands or tens of thousand iterations.

Original: For instance, for a MNIST training dataset consisting of 50,000 images, training a neural network for 100 epochs corresponds to almost 20,000 iterations with a batch size of 128.
Revised: For instance, for a MNIST training dataset consisting of 50,000 images, training a neural network for 100 epochs corresponds to almost 20,000 iterations with a batch size of 128.

Original: Hundreds of iterations are good for the ﬁrst few epochs during optimizer training, since we would like to only focus on decreasing the loss in the ﬁrst hundreds of iterations.
Revised: Hundreds of iterations are good for the ﬁrst few epochs during optimizer training, since we would like to only focus on decreasing the loss in the ﬁrst hundreds of iterations.

Original: However, as training goes on, we would like to see that the later iterations could also decrease the training loss.
Revised: However, as training goes on, we would like to see that the later iterations could also decrease the training loss.

Original: During this stage, only hundreds of iterations are clearly not enough.
Revised: During this stage, only hundreds of iterations are clearly not enough.

Original: In order to overcome this issue, we bring the idea from (Bengio et al., 2009).
Revised: In order to overcome this issue, we bring the idea from (Bengio et al., 2009).

Original: Speciﬁcally, We set a list for the number of iterations: [100, 200, 500, 1000, 1500, 2000, 2500, 3000] and we gradually increase the number of iterations following the list for every 100 epochs in optimizer training if the optimizee loss is decreasing.
Revised: Speciﬁcally, We set a list for the number of iterations: [100, 200, 500, 1000, 1500, 2000, 2500, 3000] and we gradually increase the number of iterations following the list for every 100 epochs in optimizer training if the optimizee loss is decreasing.

Original: Once the number reaches 3000, it will not change any more until the training ends.
Revised: Once the number reaches 3000, it will not change any more until the training ends.

Original: [Figure]
Revised: [Figure]

Original: F
Revised: F

Original: A BLATION S TUDY .
Revised: A BLATION S TUDY .

Original: In order to validate various design choices, we perform the ablation study as follows:
Revised: In order to validate various design choices, we perform the ablation study as follows:

Original: • B1 :
Revised: • B1 :

Original: We use the coordinate-wise gated recurrent unit (GRU) network as the optimizer architecture.
Revised: We use the coordinate-wise gated recurrent unit (GRU) network as the optimizer architecture.

Original: We train our model directly on the Bayesian loss: Eq 9 without non-Bayesian pre-training.
Revised: We train our model directly on the Bayesian loss: Eq 9 without non-Bayesian training.

Original: • B2 :
Revised: • B2 :

Original: We replace the GRU network with the LSTM network.
Revised: We replace the GRU network with the LSTM network.

Original: • BL2O : We add the non-Bayesian pre-training stage to ﬁnd a local optimum of interest  before training on the Bayesian loss.
Revised: • BL2O : We add the non-Bayesian training stage to ﬁnd a local optimum of interest ﬁrst before training on the Bayesian loss.

Original: [Figure]
Revised: [Figure]

Original: We test these three models on the Rastrigin test function.
Revised: We test these three models on the Rastrigin test function.

Original: As shown in Fig 5, B2 (LSTM) has slightly better performance in both optimization and UQ compared to GRU .
Revised: As shown in Fig 5, B2 (LSTM) has slightly better performance in both optimization and UQ compared to GRU .

Original: But BL2O has shown superior performance compared to both B1 and B2 in optimization and UQ, which clearly demonstrates the importance of the pre-training for ﬁnding an optimum of interest
Revised: But BL2O has shown superior performance compared to both B1 and B2 in optimization and UQ. These results clearly demonstrate that the two training stages must be coupled together to empower the local posterior

Original: [Figure]
Revised: [Figure]