Original: B AYESIAN L EARNING TO O PTIMIZE : Q UANTIFYING THE O PTIMIZER U NCERTAINTY
Revised: B AYESIAN L EARNING TO O PTIMIZE : Q UANTIFYING THE O PTIMIZER U NCERTAINTY

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Optimizing an objective function with uncertainty awareness is well-known to improve the accuracy and conﬁdence of optimization solutions.
Revised: Optimizing an objective function with uncertainty awareness is well-known to improve the accuracy and conﬁdence of optimization solutions.

Original: Meanwhile, another relevant but very different question remains yet open: how to model and quantify the uncertainty of an optimization algorithm itself?
Revised: Meanwhile, another relevant but very different question remains yet open: how to model and quantify the uncertainty of an optimization algorithm itself?

Original: To close such a gap, the prerequisite is to consider the optimizers as sampled from a distribution, rather than a few pre-deﬁned and ﬁxed update rules.
Revised: To close such a gap, the prerequisite is to consider the optimizers as sampled from a distribution, rather than a few pre-deﬁned and ﬁxed update rules.

Original: We ﬁrst take the novel angle to consider the algorithmic space of optimizers, each being parameterized by a neural network.
Revised: We ﬁrst take the novel angle to consider the algorithmic space of optimizers, each being parameterized by a neural network.

Original: We then propose a Boltzmann-shaped posterior over this optimizer space, and approximate the posterior locally as Gaussian distributions through variational inference.
Revised: We then propose a Boltzmann-shaped posterior over this optimizer space, and approximate the posterior locally as Gaussian distributions through variational inference.

Original: Our novel model, Bayesian learning to optimize ( BL2O ) is the ﬁrst study to recognize and quantify the uncertainty of the optimization algorithm.
Revised: Our novel model, Bayesian learning to optimize ( BL2O ) is the ﬁrst study to recognize and quantify the uncertainty of the optimization algorithm.

Original: Our experiments on optimizing test functions, energy functions in proteinprotein interactions and loss functions in image classiﬁcation and data privacy attack demonstrate that, compared to state-of-the-art methods, BL2O improves optimization and uncertainty quantiﬁcation (UQ) in aforementioned problems as well as calibration and out-of-domain detection in image classiﬁcation.
Revised: Our experiments on optimizing test functions, energy functions in proteinprotein interactions and loss functions in image classiﬁcation and data privacy attack demonstrate that, compared to state-of-the-art methods, BL2O improves optimization and uncertainty quantiﬁcation (UQ) in aforementioned problems as well as calibration and out-of-domain detection in image classiﬁcation.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Computational models of many real-world applications involve optimizing non-convex objective functions.
Revised: Computational models of many real-world applications involve optimizing non-convex objective functions.

Original: As the non-convex optimization problem is NP-hard, no optimization algorithm (or optimizer) could guarantee the global optima in general and instead, their solutions’ usefulness (sometimes based on their proximity to the optima), when the optima are unknown, can be very uncertain.
Revised: As the non-convex optimization problem is NP-hard, no optimization algorithm (or optimizer) could guarantee the global optima in general, and instead, their solutions’ usefulness (sometimes based on their proximity to the optima), when the optima are unknown, can be very uncertain.

Original: Being able to quantify such uncertainty is important to not only assessing the solution uncertainty after optimization but also enhancing the search efﬁciency during optimization.
Revised: Being able to quantify such uncertainty is important to not only assessing the solution uncertainty after optimization but also enhancing the search efﬁciency during optimization.

Original: For instance, reliable and trustworthy machine learning models demand uncertainty awareness and quantiﬁcation during training (optimizing) such models, whereas in reality deep neural networks without proper modeling of uncertainty suffer from overconﬁdence and miscalibration (Guo et al., 2017).
Revised: For instance, reliable and trustworthy machine learning models demand uncertainty awareness and quantiﬁcation during training (optimizing) such models, whereas in reality deep neural networks without proper modeling of uncertainty suffer from overconﬁdence and miscalibration (Guo et al., 2017).

Original: In another application example of protein docking, although there exists epistemic uncertainty of the objective function and the aleatoric uncertainty of the protein structure data (Cao & Shen, 2020), state-ofthe-art methods only predict several single solutions (Porter et al., 2019) without any associated uncertainty, which makes those predictions hard to interpret by the end usersof protein docking methods (such as biologists).
Revised: In another application example of protein docking, although there exists epistemic uncertainty of the objective function and the aleatoric uncertainty of the protein structure data (Cao & Shen, 2020), state-of-the-art methods only predict several single solutions (Porter et al., 2019) without any associated uncertainty, which makes those predictions hard to be interpreted by the end users (such as biologists).

Original: Various optimization methods have been proposed in response to the need of uncertainty awareness.
Revised: Various optimization methods have been proposed in response to the need of uncertainty awareness.

Original: Stochastic optimization methods like random search (Zhigljavsky, 2012), simulated annealing (Kirkpatrick et al., 1983), genetic algorithms (Goldenberg, 1989) and particle swarm optimization (Kennedy & Eberhart, 1995) injected the randomness into the algorithms in order to reduce uncertainties.
Revised: Stochastic optimization methods like random search (Zhigljavsky, 2012), simulated annealing (Kirkpatrick et al., 1983), genetic algorithms (Goldenberg, 1989) and particle swarm optimization (Kennedy & Eberhart, 1995) injected the randomness into the algorithms in order to reduce uncertainties.

Original: However, these methods do not provide the uncertainty quantiﬁcation (UQ) of solutions.
Revised: However, these methods do not provide the uncertainty quantiﬁcation (UQ) of solutions.

Original: Recently, there has been growing interests in applying inference-based methods to optimization problems (Brochu et al., 2010; Shapiro, 2000; Pelikan et al., 1999).
Revised: Recently, there have been growing interests in applying inference-based methods to optimization problems (Brochu et al., 2010; Shapiro, 2000; Pelikan et al., 1999).

Original: Generally, they transfer the uncertainties within the data and model into the ﬁnal solution by modelling the posterior distribution over the global optima.
Revised: Generally, they transfer the uncertainties within the data and model into the ﬁnal solution by modelling the posterior distribution over the global optima.

Original: For instance, Bijl et al.
Revised: For instance, Bijl et al.

Original: (2016) uses sequential Monte Carlo to approximate the distribution over the optima with Thompson sampling as the search strategy.
Revised: (2016) uses sequential Monte Carlo to approximate the distribution over the optima with Thompson sampling as the search strategy.

Original: Hern´andez-Lobato et al.
Revised: Hern´andez-Lobato et al.

Original: (2014) uses kernel approximation for modelling the posterior over the optimum under Gaussian process.
Revised: (2014) uses kernel approximation for modelling the posterior over the optimum under Gaussian process.

Original: Ortega et al.
Revised: Ortega et al.

Original: ; Cao & Shen (2020) directly model the posterior over the optimum as a Boltzmann distribution.
Revised: ; Cao & Shen (2020) directly model the posterior over the optimum as a Boltzmann distribution.

Original: They not only surpass the previous methods in accuracy and efﬁciency, but also provide easy-to-interpret uncertainty quantiﬁcation.
Revised: They not only surpass the previous methods in accuracy and efﬁciency, but also provide easy-to-interpret uncertainty quantiﬁcation.

Original: Despite progress in optimization with uncertainty-awareness, signiﬁcant open questions remain.
Revised: Despite progress in optimization with uncertainty-awareness, signiﬁcant open questions remain.

Original: Existing methods consider uncertainty either within the data or the model (including objective functions)
Revised: Existing methods consider uncertainty either within the data or the model (including objective functions)

Original: (Kendall & Gal, 2017; Ortega et al., 2012; Cao & Shen, 2020).
Revised: (Kendall & Gal, 2017; Ortega et al., 2012; Cao & Shen, 2020).

Original: However, no attention was ever paid to the uncertainty arising from the optimizer that is directly responsible for deriving the end solutions with given data and model.
Revised: However, no attention was ever paid to the uncertainty arising from the optimizer that is directly responsible for deriving the end solutions with given data and models.

Original: The optimizer is usually pre-deﬁned and ﬁxed.
Revised: The optimizer is usually pre-deﬁned and ﬁxed in the optimization algorithm space.

Original: For instance, there are several popular update rules in Bayesian optimization, such as expected improvement Vazquez & Bect (2010) or upper conﬁdence bound Srinivas et al.
Revised: For instance, there are several popular update rules in Bayesian optimization, such as expected improvement Vazquez & Bect (2010) or upper conﬁdence bound Srinivas et al.

Original: (2009), that are chosen and ﬁxed for the entire process.
Revised: (2009), that are chosen and ﬁxed for the entire process.

Original: For Bayesian neural networks training, the update rule is usually chosen off-the-shelf, such as Adam, SGD, or RMSDrop.
Revised: For Bayesian neural networks training, the update rule is usually chosen off-the-shelf, such as Adam, SGD, or RMSDrop.

Original: The uncertainty in the optimizer is intrinsically deﬁned over the optimizer space and important to uncertainty-aware optimization and solution UQ, and unwittingly ignored when the optimizer is treated as a ﬁxed sample in the space.
Revised: The uncertainty in the optimizer is intrinsically deﬁned over the optimizer space and important to the optimization and UQ solutions. However, such uncertainty is unwittingly ignored when the optimizer is treated as a ﬁxed sample in the space.

Original: To ﬁll the aforementioned gap, the core intellectual value of this work is to recognize and quantify a new form of uncertainty, that lies in the optimization algorithm (optimizer), besides the classical data- or model- based uncertainties (also known as epistemic and aleatoric uncertainties).
Revised: To ﬁll the aforementioned gap, the core intellectual value of this work is to recognize and quantify a new form of uncertainty, that lies in the optimization algorithm (optimizer), besides the classical data- or model- based uncertainties (also known as epistemic and aleatoric uncertainties).

Original: The underlying innovation is to treat an optimizer as  sample from an algorithmic space ,i.e., a distribution of optimizers, rather than one of a few hand-crafted update rules.
Revised: The underlying innovation is to treat an optimizer as a random sample from the algorithmic space , rather than one of a few hand-crafted update rules.

Original: The key enabling technique is to consider an algorithmic spaceof optimizers, each of them being parameterized by a neural network.
Revised: The key enabling technique is to consider the algorithmic space being parameterized by a neural network.

Original: We then leverage a Boltzmann-shaped posterior over the optimizers, and approximate the posterior locally as Gaussian distributions through variational inference.
Revised: We then leverage a Boltzmann-shaped posterior over the optimizers, and approximate the posterior locally as Gaussian distributions through variational inference.

Original: Our approach, Bayesian learning to optimize ( BL2O ), for the ﬁrst time addresses the modeling of the optimizer-based uncertainty.
Revised: Our approach, Bayesian learning to optimize ( BL2O ), for the ﬁrst time addresses the modeling of the optimizer-based uncertainty.

Original: Extensive experiments on optimizing test functions, energy functions in a bioinformatics application, and loss functions in the image classiﬁcation and data privacy attack demonstrate that compared to the start-of-art methods, BL2O substantially improves the performance of optimization and uncertainty quantiﬁcation, as well as calibration and out-of-domain detection in classiﬁcation.
Revised: Extensive experiments on optimizing test functions, energy functions in a bioinformatics application, and loss functions in the image classiﬁcation and data privacy attack demonstrate that compared to the start-of-art methods, BL2O substantially improves the performance of optimization and uncertainty quantiﬁcation, as well as calibration and out-of-domain detection in classiﬁcation.

Original:  R ELATED WORK
Revised:  R ELATED WORK

Original: Many works (Wang & Jegelka, 2017; Hennig & Schuler, 2012) studied optimization with uncertainty quantiﬁcation under the framework of Bayesian Optimization (Shahriari et al., 2016; Brochu et al., 2010).
Revised: Many works (Wang & Jegelka, 2017; Hennig & Schuler, 2012) studied optimization with uncertainty quantiﬁcation under the framework of Bayesian optimization (Shahriari et al., 2016; Brochu et al., 2010).

Original: In Bayesian optimization, multiple objectives are sampled from the posterior over the objectives ( p ( f |D ) ), where D is the observed data.
Revised: In these studies, multiple objectives are sampled from the posterior over the objectives ( p ( f |D ) ), where D is the observed data.

Original: Each sampled objective is optimized for obtaining samples of the global optima: w ∗ so that the empirical distribution over w ∗ can be built.
Revised: Each sampled objective is optimized for obtaining samples of the global optima: w ∗ so that the empirical distribution over w ∗ can be built.

Original: However, since this approach needs optimization for each sample, which is extreme time-consuming, signiﬁcant approximation is much needed.
Revised: Approximation is much needed since those approaches need optimization for every sample.

Original: For instance, Henr ´ andez-Lobato et al.
Revised: For instance, Henr ´ andez-Lobato et al.

Original: (2014) uses kernel approximation to approximate the posterior distirbution.
Revised: (2014) uses kernel approximation to approximate the posterior distirbution.

Original: Another line of work uses various sampling schemes for estimating the density of posterior distributions.
Revised: Another line of work uses various sampling schemes for estimating the density of posterior distributions.

Original: For instance, Bijl et al.
Revised: For instance, Bijl et al.

Original: (2016) uses sequential Monte Carlo sampling.
Revised: (2016) uses sequential Monte Carlo sampling.

Original: De Bonet et al.
Revised: De Bonet et al.

Original: (1997) designs a randomized optimization algorithm that directly samples global optima.
Revised: (1997) designs a randomized optimization algorithm that directly samples global optima.

Original: These methods are much more efﬁcient, but their performance heavily depends on the objective landscapes.
Revised: These methods are much more efﬁcient, but their performance heavily depends on the objective landscapes.

Revised: Moreover, a couple of studies (Ahmed et al., 2016; Lizotte, 2008; Osborne et al., 2009; Wu et al., 2017) in Bayesian optimization utilize ﬁrst-order information to boost the performance of optimization.

Revised: For instance, Osborne et al.

Revised: (2009) uses gradient information to improve the covariance matrix in

Original:  Finally, there are approaches (Ortega et al., 2012; Cao & Shen, 2020) that directly model the shape of posterior as the Boltzmann distributions: p ( w ∗ | D ) ∝ exp ( − αf ( w  )) , where α is the scheduled temperature constant.
Revised: Gaussian process. Wu et al. (2017) embeds the derivative knowledge into the acquisition function which is optimized in every iteration. Finally, there are approaches (Ortega et al., 2012; Cao & Shen, 2020) that directly model the shape of posterior as the Boltzmann distributions: p ( w ∗ | D ) ∝ exp( − αf ( w ∗ )) , where α is the scheduled temperature constant.

Original: The algorithm automatically adjusts α during the search in order to balance the exploration-exploitation tradeoff.
Revised: They automatically adjust α during the search in order to balance the exploration-exploitation tradeoff.

Original: Those works beat previous work in terms of both efﬁciency and accuracy.
Revised: They beat previous work in terms of both efﬁciency and accuracy.

Original: However, as revealed in the aforementioned gap, none of these methods consider the uncertainty within the optimizer.
Revised: However, as revealed in the aforementioned gap, none of these methods consider the uncertainty within the optimizer.

Original: In the following sections, we ﬁrst formally deﬁne the problem of optimization with uncertainty quantiﬁcation.
Revised: In the following sections, we ﬁrst formally deﬁne the problem of optimization with uncertainty quantiﬁcation.

Original: And then we brieﬂy describe how to address the optimizer uncertainty in classical optimizations.
Revised: And then we brieﬂy describe how to address the optimizer uncertainty in classical optimizations.

Original: Third, we propose our novel model, BL2O .
Revised: Third, we propose our novel model, BL2O .

Original: And lastly, we compare our BL2O with both Bayesian and non-Bayesian competing methods on extensive test functions and real-world applications.
Revised: And lastly, we compare our BL2O with both Bayesian and non-Bayesian competing methods on extensive test functions and real-world applications.

Original:  M ETHODS3. P ROBLEM S TATEMENT
Revised:  M ETHODS

Revised: Notation .

Original: We use a bold-faced uppercase letter to denote a matrix (e.g. W ), a bold-faced lowercase letter to denote a vector (e.g. w ), and a normal lowercase letter to denote a scalar (e.g. w ). 
Revised: We use a bold-faced uppercase letter to denote a matrix (e.g. W ), a bold-faced lowercase letter to denote a vector (e.g. w ), and a normal lowercase letter to denote a scalar (e.g. w ). 3. P ROBLEM S TATEMENT

Original: The goal of optimization is to ﬁnd the global optima for an objective function f ( w ) w.r.t.
Revised: The goal of optimization is to ﬁnd the global optimum for an objective function f ( w ) w.r.t.

Original: w :
Revised: w :

Original: [Equation]
Revised: [Equation]

Original: Such problems are usually solved by iterative optimization algorithms, whose update rules could be expressed as: w t +1 =
Revised: w ∗ is assumed unknown and treated as a random vector in this study. Once a optimizer obtains ˆ w , its estimate of w ∗ , it is important to assess the quality and the uncertainty of the solution. Considering that many real-world objective functions are nonconvex and noisy in ˆ w , solution quality is often measured by || ˆ w − w ∗ || , the proximity to the global optimum rather than that to the optimal function value.

Revised: Examples include energy functions as the objective and RMSDs as the proximity measure in protein docking (Lensink et al., 2007).

Revised: Therefore, the goal of uncertainty quantiﬁcation (UQ) here is the following:

Revised: [Equation]

Revised: where r σ is the upper bound of || ˆ w − w ∗

Revised: || at σ conﬁdence level, and D denotes samples during optimization.

Revised: Such UQ results additionally provide conﬁdence in the solution ˆ w and improve model reliability for end users.

Original:  w t + δ w t , where w t and δ w t are the parameter vector  and update vector at iteration t , respectively.
Revised: To calculate the probability deﬁned in Eq 2 and perform UQ, a direct albeit challenging way is to model the posterior over w ∗ ( p ( w ∗ |D ) ) and then sample from the posterior. When the optimizer g is regarded ﬁxed in existing literature, the posterior is actually p ( w ∗ |D , g ) . A central contribution of ours is to further consider the optimizer as a source of uncertainty, model it as a random vector in an optimizer space, and perform posterior estimation of p ( w ∗ |D ) . 3. O PTIMIZER U NCERTAINTY : A F RAMEWORK

Original: δ w t is usually a function of past gradients:
Revised: An optimizer is directly responsible for optimization and thus naturally a source of solution uncertainty. To address this often-neglected uncertainty source, we ﬁrst deﬁne the space of optimizers and then model an optimizer as a point in this space. Considering that most widely-used optimizers are iterative and using at most ﬁrst-order derivatives, we restrict the optimizer space as follows:

Original: [Equation]
Revised: Deﬁnition 3.1 ((First-order Iterative) Optimizer Space) We deﬁne a ﬁrst-order, iterative algorithmic space G , where each point g ∈ G is an iterative optimizer, that has the following mapping: g ( {∇ f ( w τ ) } tτ =1 ) = δ w t , where ∇ f ( w ) τ and δ w t are the gradient and the update vector at τ th and t th iteration, respectively.

Original: where g ( · ) is a pre-deﬁned update rule (optimizer).
Revised: Here we use g ( · ) to denote a pre-deﬁned update rule and the resulting optimizer.

Original: For instance, in gradient descent, g ( {∇ (cid:96) ( w τ ) } tτ =1 ) 
Revised: For instance, in gradient descent, g ( {∇ f ( w τ ) } tτ =1 ) =

Original: = − α ∇ (cid:96) ( w t ) , where α is the step size.
Revised: − α ∇ f ( w t ) , where α is the step size.

Original: Once we obtain the ﬁnal solution ˆ w , it is important to assess the quality of the solution as || ˆ w − w ∗
Revised: Now that the optimizer space is deﬁned, we next deﬁne the (unknown) optimal optimizer and its uncertainty.

Original: || .
Revised: Deﬁnition 3.2 (Optimal Optimizer)

Original: However, since w ∗ is unknown, instead, we can provide the following probably approximately correct assessment of ˆ w :
Revised: We deﬁne the optimal optimizer g ∗ ∈ G as the optimizer that can obtain the lowest function value with a ﬁxed budget T :

Original: [Equation]
Revised: [Equation]

Original: wherer σ is the upper bound of || ˆ w − w ∗ || at σ conﬁdence level, and D ={ w i , f ( w i ) } 
Revised: where w tg = w t − 1 g + g ( {∇ f ( w τg ) } t − 1 τ =1 ) is the parameter value at t th iteration updated through the optimizer g .

Original: Ti =1 are the observed data.
Revised: In practice the optimal optimizer g ∗ is unknown so we treat g ∗ as a random vector and formally deﬁne the optimizer uncertainty as follows: Deﬁnition 3.3 (Optimizer Uncertainty) Let G be the algorithmic space, where each point g ∈ G is an optimizer. We assume there is a prior distribution over the optimal optimizer g ∗ as p ( g ∗ ) . We also assume a likelihood distribution as p ( D| g ∗ ) , where D are the observed data (sample trajectory) given g ∗ .

Original: It is straightforward to ﬁrst model the posterior over the global optima ( p ( w ∗ |D )  ) and then sample from the posterior to obtain Eq 3. If the optimizer g is ﬁxed, then we can put g intothe condition as p ( w ∗ |D, g ) .
Revised: Then we deﬁne the optimizer uncertainty through p ( g ∗ |D ) ∝ p ( D| g ∗ ) p ( g ∗ ) . To inject the optimizer uncertainty into p ( w ∗ |D ) , it is straightforward to have the following integration for posterior estimation:

Original:  3.  O PTIMIZER U NCERTAINTY IN CLASSICAL METHODS
Revised: [Equation] 3. P ARAMETERIZING THE O PTIMIZER S PACE

Original: A simple way for considering the optimizer uncertaintyfor ﬁxed optimizers (e.g. Adam, Gradient Descent) is running algorithms for multiple times with different hyperparameters and(or) different starting points for obtaining ˆ w and modelling p ( w ∗ |D )  . 
Revised: The optimizer uncertainty p ( g ∗ |D ) as deﬁned in Def. 3.3 can be intractable when there is no proper parameterization of the optimizer space G . Therefore, we next introduce possible ways of parameterizing G as deﬁned in Def. 3.1.

Original: Such strategy can reduce the uncertainty within the optimizer to some extent. However, it heavily relies on  the pre-deﬁned distributions (discrete grids) over the hyperparameters and(or)
Revised: Parameterization through Hyperparameters of Speciﬁc Optimizers. A simple way to parameterize the optimizer space for classical optimizers (e.g. Gradient Descent, Adam) is based on their hyperparameters: G = H , where H is the hyperparameter space.

Original:  the start points.
Revised: For instance, for gradient descent, we have H = ( α ) , where α is the learning rate. For Adam, we have H = ( α, β 1 , β 2 ) , where β 1 and β 2 are the coefﬁcients used for computing running averages of gradient and its square.

Original: More importantly, an optimizer cannot be just represented by its hyperparameters or starting points. The intrinsic uncertainty that lies in its update rule is not explored at all in this strategy.
Revised: However, such parameterization has signiﬁcant drawbacks. The resulting algorithmic space G is very limited and heavily depends on the speciﬁc optimizer. The G (a 1D space) parameterized by the hyperparameters of gradient descent is different from that (a 3D space) parameterized by the hyperparameters of Adam.

Revised: In fact, each is a rather restricted region of the actual G .

Revised: The intrinsic ﬂexibility (uncertainty) that lies in an iterative optimizer’s update rule is not explored at all in this parameterization.

Original: Such statements can be clearly demonstratedby extensive experiments in Sec. 4. 
Revised: These drawbacks are empirically demonstrated in Sec. 4. Parameterization through Neural Networks.

Original: In order to consider the intrinsic uncertainty within the update rule, we assume that there exists an algorithm space  G  where each point g ∈ G is an optimizer.
Revised: In order to reasonably and accurately model the intrinsic uncertainty within the update rule, we need to ﬁnd a much more ﬂexible way for modelling g . We thus consider to parameterize the optimizer space as a neural network: G = Θ , where each θ ∈ Θ are the parameters in the neural network.

Original: In order to quantify the uncertainty over the optimizer, a good way is to ﬁrst model the posterior distribution over G as p ( g ( · ) |D ) . And then integrate this posterior into the ﬁnal posterior through:
Revised: Overcoming drawbacks of the optimizer space H by hyperparameters, Θ by neural network parameters generalizes update rules through neural networks that can represent a wide variety of functions. We note that this is also the space of meta-optimizers that learn to optimize (L2O) iterative update rules from data on a given task (Andrychowicz et al., 2016; Chen et al., 2017; Lv et al., 2017; Cao et al., 2019a).

Original: [Equation] However,modelling p ( g ( · ) |D ) seems intractable as there is no parameterization of  the algorithmic space.
Revised: However, there has been no notion of uncertainty let alone the task of UQ for the learned optimizer in these L2O methods, which is to be addressed in our Bayesian L2O (BL2O).

Original: In the next subsection, we will parameterize this algorithmic space by the parameters of a neural network so that modelling p ( g ( · ) |D ) becomes feasible. 3. P ARAMETERIZING THE SPACE THROUGH A N EURAL N ETWORK
Revised: Now that we have the optimizer space G properly deﬁned and parameterized, we proceed to model an optimizer g as a random vector in the space.

Original: Boltzmann-shaped posterior
Revised: Boltzmann-shaped Posterior.

Original: In order to reasonably and feasibly model the intrinsic uncertainty within the optimizer g ( · ) , we parameterize g ( · ) asa neural network: g θ ( · ) , where θ are the parameters in the neural network.
Revised: Since we have G = Θ , we can rewrite each g ∈ G as g θ with θ ∈ Θ and the optimal optimizer g ∗ as g θ ∗ .

Original: Then p ( g ( · ) |D ) becomes p ( θ  |D ) .
Revised: Therefore, p ( g ∗ |D ) becomes p ( θ ∗ |D ) .

Original: We use the LSTM architecture from Andrychowicz et al.
Revised: We consider a Gaussian prior over the parameters of the neural network : p ( θ ∗ ) ∝ exp( − λ ||

Original: as our optimizer’s architecture. To make g θ ( · ) has good optimization performance, it is important to optimize the following loss:
Revised: θ ∗ || 22 ) , where λ is a constant controlling the variance. We use the chain rule to decompose the likelihood function p ( D| θ ∗ ) at a ﬁxed budge T : [Equation]

Original: [Equation]
Revised: [Equation]

Original: where f 1 ( · ) , f 2 ( · ) , ..., f N ( · ) are N objective functions used for training the neural network; ˆ w j is the ﬁnal solution of applying g θ ( · ) on j th objective and λ is the L2 regularization constant.
Revised: The second equality is due to that w t θ ∗ is ﬁxed given θ ∗ and past data points.

Original: If the training objective f j ( · ) is the loss function of a neural network classiﬁcation task, then for the j th mini-batch samples consisting of { x ij , y ij } Mi =1 , we have the following training objective assuming independence among samples:
Revised: For the single sample likelihood, we apply the results from Ortega et al.

Original: [Equation]
Revised: [Equation]

Original: Then we can re-write Eq 5 as:
Revised: ; Cao & Shen (2020) and obtain

Original: [Equation]
Revised: [Equation]

Original: We consider the ﬁrst term of Eq 7 as the negative log likelihood term, while the second term as the negative log-prior term. Then we could regard F ( θ ) as the negative logarithm of un-normalized posterior distribution over θ : F (  θ  ) ∝  − log p (  θ |D ) .
Revised: We multiply the likelihood functions of all samples together and obtain the Boltzmann-shaped likelihood function as p ( D| θ ∗ ) ∝ exp( − (cid:80) Tt =1 f ( w t θ ∗ )) .

Original: We then obtain the  posterior as the following Boltzmann distribution:
Revised: We ﬁnally multiply the conjugate prior to the likelihood and obtain the Boltzmann-shaped posterior as:

Original: [Equation]
Revised: [Equation]

Original: In the general case of optimizing a function f ( · ) besides classiﬁcation loss, we use a Boltzmannshaped posterior where F ( θ  ) is as deﬁned in Eq.
Revised: where F ( θ ∗ ) = (cid:80) Tt =1 f ( w t θ ∗ ) + λ

Original: Similar ideas were proposed in Ortega et al. ; Cao & Shen (2020).
Revised: || θ ∗ || 22 , which actually contains the objective in Eq 3 plus a L regularization constant.

Original: Local Approximation and Bayesian Loss
Revised: Local Approximation and Bayesian Loss.

Original: However, the above posterior distribution involves an integral in the normalization constant which is computationally intractable.
Revised: However, the above posterior distribution involves an integral in the normalization constant which is computationally intractable.

Original: Moreover, the architecture of F ( θ  ) is so complicated that it is impossible to directly sample from the posterior distribution.
Revised: Moreover, the architecture of F ( θ ∗ ) is so complicated that it is impossible to directly sample from the posterior distribution.

Original: In order to overcome the aforementioned challenges, we would like to learn a distribution function q ( θ  | φ ) that has the analytic form and is easy to be sampled, where φ is the parameter vector in q ( θ  | φ ) , to approximate the real posterior p ( θ  |D ) .
Revised: In order to overcome the aforementioned challenges, we would like to learn a distribution function q ( θ ∗ | φ ) that has the analytic form and is easy to be sampled, where φ is the parameter vector in q ( θ ∗ | φ ) , to approximate the real posterior p ( θ ∗ |D ) .

Original: Furthermore, due to the high dimensions of θ  and the complicated landscape of the posterior, it is impossible to approximate p ( θ  |D ) at every position in the θ  space.
Revised: Furthermore, due to the high dimensions of θ ∗ and the complicated landscape of the posterior, it is impossible to approximate p ( θ ∗ |D ) at every position in the θ ∗ space.

Original: We then consider to approximate it locally around θ ∗ , an optimum of interest for F ( θ  ) .
Revised: We then consider to approximate it locally around θ c , an optimum of interest for F ( θ ∗ ) .

Original: We denote the local region as Θ ∗ , a neighborhood around θ ∗ , and re-normalization constant C = (cid:82) θ  ∈
Revised: We denote the local region as Θ c , a neighborhood around θ c , and re-normalization constant C = (cid:82) θ ∗ ∈

Original: Θ ∗ p ( θ  |D )
Revised: Θ c p ( θ ∗ |D )

Original: d θ  .
Revised: d θ ∗ .

Original: Then the local posterior will be a conditioned (re-scaled) version of p ( θ  |D ) : 
Revised: Then the local posterior will be a conditioned (re-scaled) version of p ( θ ∗ |D ) : p (cid:48) ( θ ∗ |D )

Original: [Equation]
Revised: = p ( θ ∗ |D ) /C, θ ∗ ∈ Θ c .

Original: In order to make q ( θ  | φ ) ≈ p (cid:48) ( θ  |D ) , we calculate the KL-divergence between these two:
Revised: In order to make q ( θ ∗ | φ ) ≈ p (cid:48) ( θ ∗ |D ) , we calculate the KL-divergence between these two:

Original: [Equation]
Revised: [Equation]

Original: where Z = (cid:82) exp( − F ( θ  ))
Revised: where Z = (cid:82) exp( − F ( θ ∗ ))

Original: d θ  is the normalization constant.
Revised: d θ ∗ is the normalization constant.

Original: The second term in the above equation equals to log( ZC ) , a constant w.r.t.
Revised: The second term in the above equation equals to log( ZC ) , a constant w.r.t.

Original: φ , thus could be ignored during optimization.
Revised: φ , thus could be ignored during optimization.

Original: We then propose our Bayesian loss as:
Revised: We then propose our Bayesian loss as:

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: where the ﬁrst term of F B measures the negative entropy of our approximated posterior, and the second term is the expectation of the loss function over of posterior.
Revised: where the ﬁrst term of F B measures the negative entropy of our approximated posterior, and the second term is the expectation of the loss function over of posterior.

Original: Gaussian Posterior We consider φ = ( µ , Σ ) and q ( θ  | φ ) =
Revised: Gaussian Posterior. For local approximation, we consider φ = ( µ , Σ ) and q ( θ ∗ | φ ) =

Original: N ( µ , Σ ) , where µ is the mean vector and Σ is the covariance matrix of a normal distribution.
Revised: N ( µ , Σ ) , where µ is the mean vector and Σ is the covariance matrix of a normal distribution.

Original: For simplicity, we consider Σ to be a diagonal matrix: Σ = diag ( σ 21 , σ 22 , σ 23 , ... ) .
Revised: For simplicity, we consider Σ to be a diagonal matrix: Σ = diag ( σ 21 , σ 22 , σ 23 , ... ) .

Original: The second term in Eq (11) involves the integral over F ( θ  ) , which is intractable.
Revised: The second term in Eq (9) involves the integral over F ( θ ∗ ) , which is intractable.

Original: Therefore, we use Monte Carlo sampling through q ( θ  | φ ) to replace the integral there.
Revised: Therefore, we use Monte Carlo sampling through q ( θ ∗ | φ ) to replace the integral there.

Original: However, the direct sampling of the posterior parameters makes it difﬁcult for the optimization as it is inaccessible to get the gradient w.r.t.
Revised: However, the direct sampling of the posterior parameters makes it difﬁcult for the optimization as it is inaccessible to get the gradient w.r.t.

Original: µ and Σ .
Revised: µ and Σ .

Original: Moreover, the standard deviation σ 1 , σ 2 , ... must be non-negative, making the optimization constrained.
Revised: Moreover, the standard deviation σ 1 , σ 2 , ... must be non-negative, making the optimization constrained.

Original: To overcome those two challenges, we use the trick introduced in (Blundell et al., 2015) to shift sampling from q ( θ  | φ ) to sampling from a standard normal distribution N ( 0 , I ) .
Revised: To overcome those two challenges, we use the trick introduced in (Blundell et al., 2015) to shift sampling from q ( θ ∗ | φ ) to sampling from a standard normal distribution N ( 0 , I ) .

Original: And we reparameterize standard deviation σ i to ρ i as σ i 
Revised: And we reparameterize standard deviation σ i to ρ i as σ i =

Original: [-= log(
 + exp(-] ρ i )) .
Revised: [-= log(
 + exp(-]log(1+exp( ρ i )) .

Original: Then for any (cid:15) sampled from N ( µ , I ) , we could calculate θ  as θ  =u
Revised: Then for any u sampled from N ( µ , I ) , we could calculate θ ∗ as θ ∗ =

Original:  + log(1 + exp( ρ )) , where ρ = ( ρ 1 , ρ 2 , ... ) .
Revised: u + log(1 + exp( ρ )) , where ρ = ( ρ 1 , ρ 2 , ... ) .

Original: 3. B AYESIAN A VERAGING
Revised: 3. B AYESIAN A VERAGING

Original: We recall our goal to build the posterior over the global optimum: p ( w ∗ |D ) through Eq 4.
Revised: We recall our goal to build the posterior over the global optimum: p ( w ∗ |D ) through Eq 4.

Original: We consider using Monte Carlo sampling to approximate the integral as:
Revised: We consider using Monte Carlo sampling to approximate the integral as:

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: where θ  i is sampled from q ( θ  | φ )and N = 10 , 000 .
Revised: where θ ∗ i is sampled from q ( θ ∗ | φ ) .

Revised: Since q ( θ ∗ | φ ) follows a multivariate Gaussian distribution where individual dimensions are independent of each other, we estimate the summation above in each dimension using independent MC samplings.

Revised: In practice, N being 10 , 000 , 100 , 000 , and 500 , 000 led to negligible differences in the 1D estimations, and N was thus ﬁxed at 10 , 000 .

Original: 3.T RAINING S TAGES AND M ODEL I MPLEMENTATION
Revised: 3. M ETA - TRAINING S ET

Revised: In order to boost the robustness and generalizability of our optimizer posterior p ( g ∗ |D ) , we consider using an ensemble of objective functions F = { f i } Ni =1 .

Revised: Speciﬁcally, we replace the objective function in Eq 3 with 1 N (cid:80) Ni =1 (cid:80)

Revised: Tt =

Revised: 1 f i ( w t θ ∗ ,i ) and rewrite F ( θ ∗ ) as:

Revised: [Equation]

Revised: where w t θ ∗ ,i is the solution at t th iteration for objective f i optimized by g θ ∗ .

Revised: Such replacement can let our posterior generalize to novel objective functions.

Revised: We regard the functional dataset F as the meta-training set.

Revised: During the experiments, we create different meta-training sets for different problems, which will be described in details in Sec 4.

Revised: 3. T RAINING S TAGES , M ODEL A RCHITECTURE AND I MPLEMENTATION

Original: As mentioned before, our posterior is local around θ ∗ , an optimum of interest.
Revised: As mentioned before, our posterior is local around θ c , an optimum of interest.

Original: To obtain θ ∗ , we ﬁrst pre-train our model in a non-Bayesian way through optimizing the loss in Eq 5.
Revised: To obtain θ c , we ﬁrst pre-train our model in a non-Bayesian way through minimizing the loss in Eq 11.

Original: We then use θ ∗ as the warm start for µ , and start the second Bayesian training stage.
Revised: We then use θ c as the warm start for µ , and start the second Bayesian training stage.

Original: The model is implemented in Tensorﬂow 1.13 (Abadi et al., 2016) and optimized by Adam (Kingma & Ba, 2014).
Revised: The model is implemented in Tensorﬂow 1.13 (Abadi et al., 2016) and optimized by Adam (Kingma & Ba, 2014).

Original: Due to the coordinate-wise LSTM (Andrychowicz et al., 2016), our BL2O model only contains 10,282 free parameters.
Revised: For the optimizer architecture, we use the coordinate-wise LSTM from Andrychowicz et al. Due to the coordinate-wise nature, our BL2O model only contains 10,282 free parameters.

Original: For all experiments, the length of LSTM is set to be 20.
Revised: For all experiments, the length of LSTM is set to be 20.

Original: Both training stages include 5,000 training epochs.
Revised: Both training stages include 5,000 training epochs.

Original: 3. C OMPUTATIONAL C OMPLEXITY
Revised: 3. C OMPUTATIONAL C OMPLEXITY

Original: The time complexity for BL2O is O ( KBN e + KN e H 2 ) , where K is the number of sampling trajectories, B is the minibatch size, N e is the number of objective parameters, and H is the hidden size of LSTM ( H = 20 in the study).
Revised: The time complexity for BL2O is O ( KBN e + KN e H 2 ) , where K is the number of sampling trajectories, B is the minibatch size, N e is the number of objective parameters, and H is the hidden size of LSTM ( H = 20 in the study).

Original: As the batch size increases, the computational cost is close to the traditional Bayesian neural networks trained through SGD.
Revised: As the batch size increases, the computational cost is close to the traditional Bayesian neural networks trained through SGD.

Original: Due to the coordinate-wise LSTM, the space coomplexity (memory cost) of BL2O is only O
Revised: Due to the coordinate-wise LSTM, the space coomplexity (memory cost) of BL2O is only O

Original: ( H 2 ) , which remains the same as the number of objective parameters varies.
Revised: ( H 2 ) , which remains the same as the number of objective parameters varies.

Original: Both the time and the space complexity of BL2O are the same as
Revised: Both the time and the space complexity of BL2O are the same as

Original: DM LSTM (Abadi et al., 2016), while those of Adam are O ( KBN e ) and O ( N e ) , respectively.
Revised: DM LSTM (Abadi et al., 2016), while those of Adam are O ( KBN e ) and O ( N e ) , respectively.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: We test our BL2O model extensively on optimizing: non-convex test functions, energy functions in protein-protein interactions, loss functions in image classiﬁcation and loss functions in data privacy attack.
Revised: We test our BL2O model extensively on optimizing: non-convex test functions, energy functions in protein-protein interactions, loss functions in image classiﬁcation and loss functions in data privacy attack.

Original: We compare BL2O to three non-Bayesian methods: Adam, Particle Swarm Optimization (PSO) (Kennedy & Eberhart, 1995), DM LSTM (Andrychowicz et al., 2016) and a recently published Bayesian method, BAL (Cao & Shen, 2020).
Revised: We compare BL2O to three non-Bayesian methods: Adam, Particle Swarm Optimization (PSO) (Kennedy & Eberhart, 1995), DM LSTM (Andrychowicz et al., 2016) and a recently published Bayesian method, BAL (Cao & Shen, 2020).

Original: All algorithms are running for 10,000 times with random initializing points to obtain the empirical posterior distributions.
Revised: All algorithms are running for 10,000 times with random initializing points to obtain the empirical posterior distributions.

Original: During each run, the hyperparameters in Adam and PSO are sampled from Table 4 in Appendix A.
Revised: During each run, the hyperparameters in Adam and PSO are sampled from Table 4 in Appendix A.

Original: Out of 10,000 solutions we choose the one with the lowest function value to be the ﬁnal solution ( ˆ w ).
Revised: Out of 10,000 solutions we choose the one with the lowest function value to be the ﬁnal solution ( ˆ w ).

Original: Generally, for optimization performance, we assess the distance between the ﬁnal solution and the global optima:
Revised: Generally, for optimization performance, we assess the distance between the ﬁnal solution and the global optima:

Original: ||
Revised: ||

Original: ˆ w − w ∗
Revised: ˆ w − w ∗

Original: || .
Revised: || .

Original: The lower the distance is, the better the solution quality is.
Revised: The lower the distance is, the better the solution quality is.

Original: For uncertainty quantiﬁcation, we assess the upper bound r σ and the real conﬁdence (cid:15) σ given a ﬁxed conﬁdence level σ .
Revised: For uncertainty quantiﬁcation, we assess the upper bound r σ and the real conﬁdence (cid:15) σ given a ﬁxed conﬁdence level σ .

Original: The real conﬁdence (cid:15) σ is deﬁned by the fraction of 10,000 solutions that actually fall in the bounded region.
Revised: The real conﬁdence (cid:15) σ is deﬁned by the fraction of 10,000 solutions that actually fall in the bounded region.

Original: The lower the r σ is, the tighter the conﬁdence interval is.
Revised: The lower the r σ is, the tighter the conﬁdence interval is.

Original: And the closer of (cid:15) σ to σ is, the more accurate the conﬁdence estimate is.
Revised: And the closer of (cid:15) σ to σ is, the more accurate the conﬁdence estimate is.

Original: Comparison in optimizing test functions.
Revised: Comparison in optimizing test functions.

Original: We ﬁrst test the performance on test functions in the global optimization benchmark set (Jamil & Yang, 2013).
Revised: We ﬁrst test the performance on test functions in the global optimization benchmark set (Jamil & Yang, 2013).

Original: We choose three extremely rugged, nonconvex functions: Rastrigin, Ackley and Griewank in 5 dimensions: 6D, 12D, 18D, 24D, 30D. For each function, we create a diverse, broad family of similar functions f j ( w ) as the meta-training set used for training DM LSTM and BL2O. The analytical forms and the meta-training sets of those functions are shown in Table 5 in Appendix B.
Revised: We choose three extremely rugged, nonconvex functions: Rastrigin, Ackley and Griewank in 5 dimensions: 6D, 12D, 18D, 24D, 30D. For each function, we create a diverse, broad family of similar functions f j ( w ) as the meta-training set used for training DM LSTM and BL2O. The analytical forms and the meta-training sets of those functions are shown in Table 5 in Appendix B.

Original: We compare BL2O with all 4 competing methods.
Revised: We compare BL2O with all 4 competing methods.

Original: The optimization and UQ performances are shown in Fig.
Revised: The optimization and UQ performances are shown in Fig.

Original: In all three cases and 5 dimensions, BL2O has led to the best solution quality.
Revised: In all three cases and 5 dimensions, BL2O has led to the best solution quality.

Original: In terms of UQ, BL2O has shown the most accurate conﬁdence estimation ( (cid:15) σ ≈ σ ) when σ = 0 .
Revised: In terms of UQ, BL2O has shown the most accurate conﬁdence estimation ( (cid:15) σ ≈ σ ) when σ = 0 .

Original: 9 and σ = 0 .
Revised: 9 and σ = 0 .

Original: 8 , while BAL was the second best.
Revised: 8 , while BAL was the second best.

Original: And BL2O has shown much tighter conﬁdence intervals r σ against BAL.
Revised: And BL2O has shown much tighter conﬁdence intervals r σ against BAL.

Original: In some cases, although DM LSTM has lower r σ than BL2O, it has much lower conﬁdence level, indicating that this tight upper bound in DM LSTM is miscalibrated.
Revised: In some cases, although DM LSTM has lower r σ than BL2O, it has much lower conﬁdence level, indicating that this tight upper bound in DM LSTM is miscalibrated.

Original: As a result,
Revised: As a result,

Original: BL2O has shown the best performance in both optimization and UQ.
Revised: BL2O has shown the best performance in both optimization and UQ.

Original: [Figure] ̂ [Figure] |
Revised: We also perform the ablation study on the Rastrigin function for validating various design choices in Appendix F.

Original: [Figure]
Revised: [Figure]

Original: Comparison in optimizing energy functions for protein docking.
Revised: Comparison in optimizing energy functions for protein docking.

Original: We then apply BL2O to a bioinformatics application: predicting the 3D structures of protein-complexes (Smith & Sternberg, 2002), called protein docking.
Revised: We then apply BL2O to a bioinformatics application: predicting the 3D structures of protein-complexes (Smith & Sternberg, 2002), called protein docking.

Original: Ab initio protein docking can be recast as optimizing a noisy and expensive energy function in a high-dimensional conformational space (Cao & Shen, 2020): x ∗ = arg min x f ( x ) .
Revised: Ab initio protein docking can be recast as optimizing a noisy and expensive energy function in a high-dimensional conformational space (Cao & Shen, 2020): x ∗ = arg min x f ( x ) .

Original: While solving such optimization problems still remains difﬁcult, quantifying the uncertainty of resulting optima (docking solutions) is even more challenging.
Revised: While solving such optimization problems still remains difﬁcult, quantifying the uncertainty of resulting optima (docking solutions) is even more challenging.

Original: In this section, we apply our BL2O to optimization and uncertainty quantiﬁcation in protein docking and compare with a state-of-the-art method BAL (Cao & Shen, 2020).
Revised: In this section, we apply our BL2O to optimization and uncertainty quantiﬁcation in protein docking and compare with a state-of-the-art method BAL (Cao & Shen, 2020).

Original: We describe the detailed settings of BL2O on protein docking in Appendix C. From BL2O, we obtain a posterior distribution p ( w ∗ | D ) over the native structure w ∗ and the lowest energy structure, ˆ w .
Revised: We describe the detailed settings of BL2O on protein docking in Appendix C. From BL2O, we obtain a posterior distribution p ( w ∗ | D ) over the native structure w ∗ and the lowest energy structure, ˆ w .

Original: In protein docking, the quality of a predicted structure is based on the distance to the native structure (the global optimum): || ˆ w − w ∗
Revised: In protein docking, the quality of a predicted structure is based on the distance to the native structure (the global optimum): || ˆ w − w ∗

Original: || .
Revised: || .

Original: For UQ, we assess the two-sided conﬁdence interval at σ = 0 .as
Revised: For UQ, we assess the two-sided conﬁdence interval at σ = 0 .as

Original: P ( l 0 .
Revised: P ( l 0 .

Original: 9 (cid:54) ||
Revised: 9 (cid:54) ||

Original: ˆ w − w ∗
Revised: ˆ w − w ∗

Original: ||
Revised: ||

Original: (cid:54) r 0 . 9 )
Revised: (cid:54) r 0 . 9 )

Original: = 0 .
Revised: = 0 .

Original: In Table 1, we assess ||
Revised: In Table 1, we assess ||

Original: ˆ w − w ∗
Revised: ˆ w − w ∗

Original: || , r 0 .
Revised: || , r 0 .

Original: 9 − l 0 .
Revised: 9 − l 0 .

Original: 9 and whether || ˆ w − w ∗
Revised: 9 and whether || ˆ w − w ∗

Original: || is within the conﬁdence interval.
Revised: || is within the conﬁdence interval.

Original: For optimization, BL2O clearly outperforms BAL in two medium cases while performing slightly worse in the other cases.
Revised: For optimization, BL2O clearly outperforms BAL in two medium cases while performing slightly worse in the other cases.

Original: Yet for UQ, BL2O shows clearly superior performance over BAL in all cases, with accurate or/and tight conﬁdence intervals.
Revised: Yet for UQ, BL2O shows clearly superior performance over BAL in all cases, with accurate or/and tight conﬁdence intervals.

Original: We also visulize the posterior distributions over || ˆ w − w ∗ || for protein 1JMO 4.
Revised: We also visulize the posterior distributions over || ˆ w − w ∗ || for protein 1JMO 4.

Original: As shown in Fig 2, we can see compared to that of BAL,
Revised: As shown in Fig 2, we can see compared to that of BAL,

Original: BL2O’s distribution has real ||
Revised: BL2O’s distribution has real ||

Original: ˆ w − w ∗ || within the 90% C.I. and smaller variance.
Revised: ˆ w − w ∗ || within the 90% C.I. and smaller variance.

Original: More posterior distributions are shown in Appendix D.
Revised: More posterior distributions are shown in Appendix D.

Original: [Table]
Revised: [Table]

Revised: Target (docking diffﬁculty

Original: [Figure]
Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Original:  Comparison in optimizing loss functions in image classiﬁcation.
Revised: [Figure] [Table] [Figure] Comparison in optimizing loss functions in image classiﬁcation.

Original: We then test the performance of optimizing the loss function in image classiﬁcation on the MNIST dataset.
Revised: We then test the performance of optimizing the loss function in image classiﬁcation on the MNIST dataset.

Original: We apply a 2-layers MLP network as the classiﬁer.
Revised: We apply a 2layers MLP network as the classiﬁer.

Original: The competing methods include Adam, DM LSTM and a Bayesian neural network method: variational inference (VI) (Blundell et al., 2015).
Revised: The competing methods include Adam, DM LSTM and two Bayesian neural network methods: variational inference (VI) (Blundell et al., 2015) and Learnable

Original:  Moreover, for DM LSTM and BL2O, we apply a trick during the optimizer training called curriculum learning (CL) and introduce it in detail in Appendix E for training over long-term iterations.
Revised: Bernoulli Dropout (LBO) (Boluki et al., 2020). Moreover, for DM LSTM and BL2O, we apply a trick during the optimizer training called curriculum learning (CL) and introduce it in detail in Appendix E for training over long-term iterations.

Original: We call DM LSTM with CL as DM LSTM C and BL2O with CL as BL2O C.
Revised: We call DM LSTM with CL as DM LSTM C and BL2O with CL as BL2O C.

Original: The assessment of the optimization and UQ for this machine learning task is different from that for optimization before.
Revised: The assessment of the optimization and UQ for this machine learning task is different from that for optimization before.

Original: In terms of optimization, we assess the classiﬁcation accuracy on the test set.
Revised: In terms of optimization, we assess the classiﬁcation accuracy on the test set.

Original: In terms of UQ, we measure two metrics that assess the robustness and trustworthiness of the classiﬁer: the in-domain calibration error and the out-of-domain detection rate.
Revised: In terms of UQ, we measure two metrics that assess the robustness and trustworthiness of the classiﬁer: the in-domain calibration error and the out-of-domain detection rate.

Original: We ﬁrst compare the accuracy on the testing set among different methods.
Revised: We ﬁrst compare the accuracy on the testing set among different methods.

Original: As shown in Table 2,
Revised: As shown in Table 2,

Original: Adam, DM LSTM C and BL2O C have almost the same best performance.
Revised: Adam, DM LSTM C and BL2O C have almost the same best performance.

Original: The signiﬁcant improvement from DM LSTM to DM LSTM C, and from BL2O to BL2O C shows the big advantage of curriculum learning in learning to optimize.
Revised: The signiﬁcant improvement from DM LSTM to DM LSTM C, and from BL2O to BL2O C shows the big advantage of curriculum learning in learning to optimize.

Original: In conclusion, BL2O C had on par accuracy with
Revised: In conclusion, BL2O C had on par accuracy with

Original: Adam and DM LSTM C on the MNIST dataset.
Revised: Adam and DM LSTM C on the MNIST dataset.

Original: However, classiﬁcation models must not only be accurate, but also indicate when they are likely to be incorrect.
Revised: However, classiﬁcation models must not only be accurate, but also indicate when they are likely to be incorrect.

Original: Conﬁdence calibration, the probability that estimates the true likelihood of each prediction is also important for classiﬁcation models.
Revised: Conﬁdence calibration, the probability that estimates the true likelihood of each prediction is also important for classiﬁcation models.

Original: In the ideal case, the maximum output probability (MaxConﬁdence) for each test sample should be equal to the prediction accuracy for that sample.
Revised: In the ideal case, the maximum output probability (MaxConﬁdence) for each test sample should be equal to the prediction accuracy for that sample.

Original: To assess the calibration of each methods, we split the test set into 20 equal-sized bins and assess the calibration error as the average discrepancy between accuracy and MaxConﬁdence in each bin.
Revised: To assess the calibration of each methods, we split the test set into 20 equal-sized bins and assess the calibration error as the average discrepancy between accuracy and MaxConﬁdence in each bin.

Original: As seen in Table 2, among all methods compared, BL2O C and BL2O had the least calibration error.
Revised: As seen in Table 2, among all methods compared, BL2O C and BL2O had the least calibration error.

Original: The ﬁgure of Acc. vs MaxConf. is also shown in Fig.
Revised: The ﬁgure of Acc. vs MaxConf. is also shown in Fig.

Original: 4 in Appendix E.
Revised: 4 in Appendix E.

Original: We also inspect the out-of-domain detection of BL2O, BL2O C and competing methods.
Revised: We also inspect the out-of-domain detection of BL2O, BL2O C and competing methods.

Original: We train all models on the data belonging to the ﬁrst 5 classes in the MNIST training dataset (the last layer of the optimizee is modiﬁed to have 5 rather than 10 neurons) and test them on the remaining samples from the other 5 classes.
Revised: We train all models on the data belonging to the ﬁrst 5 classes in the MNIST training dataset (the last layer of the optimizee is modiﬁed to have 5 rather than 10 neurons) and test them on the remaining samples from the other 5 classes.

Original: An ideal model would predict a uniform distribution over the 5 wrong classes.
Revised: An ideal model would predict a uniform distribution over thewrong classes.

Original: [Table] Therefore, we deﬁne the out-of-domain detection rate at threshold t , q t , as the percentage of test samples with max class conﬁdence below t .
Revised: Therefore, we deﬁne the out-of-domain detection rate at threshold t , q t , as the percentage of test samples with max class conﬁdence below t .

Original: The larger the q t , the better out-of-domain detection is.
Revised: The larger the q t , the better out-ofdomain detection is.

Original: As shown in Table 2, BL2O and BL2O C shows superior performance with all  competing methods.
Revised: As shown in Table 2, BL2O and BL2O C shows superior performance with all [Table] competing methods.

Revised: Notably, BL2O without curriculum learning had much better out-of-domain detection rates compared to BL2O with curriculum learning.

Original: Comparison in optimizing loss functions for data privacy attack.
Revised: Comparison in optimizing loss functions for data privacy attack.

Original: We ﬁnally apply our model to an application that critically needs UQ.
Revised: We ﬁnally apply our model to an application that critically needs UQ.

Original: As many machine learning models are deployed publicly, it is important to avoid leaking private sensitive information, such as ﬁnancial data, health data and so on.
Revised: As many machine learning models are deployed publicly, it is important to avoid leaking private sensitive information, such as ﬁnancial data, health data and so on.

Original: Data privacy attack (Nasr et al., 2018) studies this problem by playing the role of hackers and attacking the machine-learning models to quantify the risk of privacy leakage.
Revised: Data privacy attack (Nasr et al., 2018) studies this problem by playing the role of hackers and attacking the machine-learning models to quantify the risk of privacy leakage.

Original: Better attacks would help models to be better prepared for privacy defense.
Revised: Better attacks would help models to be better prepared for privacy defense

Original:  We use the model and dataset in (Cao et al., 2019), where each input has 9 features involving patient genetic information and the output p is the probability of the clinical signiﬁcance (having cancer or not) for a patient.
Revised: [Table] We use the model and dataset in (Cao et al., 2019b), where each input has 9 features involving patient genetic information and the output p is the probability of the clinical signiﬁcance (having cancer or not) for a patient.

Original: We study the following model inversion attack (Fredrikson et al., 2015): by giving 5 features w (cid:48) ∈ [0 , 1] 5 out of 9 and the label p of each patient, we want to recover the rest 4 features w ∗ ∈ [0 , 1] 4 (potentially sensitive patient information).
Revised: We study the following model inversion attack (Fredrikson et al., 2015): by giving 5 features w (cid:48) ∈ [0 , 1] 5 out of 9 and the label p of each patient, we want to recover the rest 4 features w ∗ ∈ [0 , 1] 4 (potentially sensitive patient information).

Original: Therefore, for each patient, the objective is
Revised: Therefore, for each patient, the objective is

Original: [Equation]
Revised: [Equation]

Original: (13) where w ∗ is the ground-truth of w and m is the trained predictive model.
Revised: (12) where w ∗ is the ground-truth of w and m is the trained predictive model.

Original: The closeness between the predicted and the real input features can quantify the risk of information leakage and the quality of the attack.
Revised: The closeness between the predicted and the real input features can quantify the risk of information leakage and the quality of the attack.

Original: We compare BL2O with Adam, PSO, BAL and DM LSTM on optimization and UQ on all test cases in (Cao et al., 2019).
Revised: We compare BL2O with Adam, PSO, BAL and DM LSTM on optimization and UQ on all test cases in (Cao et al., 2019b).

Original: The meta-training objectives for BL2O and DM LSTM are the training set in (Cao et al., 2019).
Revised: The meta-training objectives for BL2O and DM LSTM are the training set in (Cao et al., 2019b).

Original: As shown in Table 3, BL2O has shown the best performance in both optimization and UQ compared to all competing methods.
Revised: As shown in Table 3, BL2O has shown the best performance in both optimization and UQ compared to all competing methods.

Original: It is noteworthy that learned optimizers (DM LSTM and BL2O) had much better optimization performance than pre-deﬁned optimizers.
Revised: It is noteworthy that learned optimizers (DM LSTM and BL2O) had much better optimization performance than pre-deﬁned optimizers.

Original: And the Bayesian methods (BAL and BL2O) had signiﬁcantly better UQ performance than non-Bayesian methods.
Revised: And the Bayesian methods (BAL and BL2O) had signiﬁcantly better UQ performance than non-Bayesian methods.

Original: BL2O possessed the advantages of both learned and Bayesian optimizers to achieve the best performance.
Revised: BL2O possessed the advantages of both learned and Bayesian optimizers to achieve the best performance.

Original: [Table]
Revised: [Table]

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: Current optimization algorithms, even with uncertainty-awareness, do not address the uncertainty arising within the optimizer itself.
Revised: Current optimization algorithms, even with uncertainty-awareness, do not address the uncertainty arising within the optimizer itself.

Original: To close this gap, we parameterize the update rule as a neural network and build a Boltzmann-shaped posterior over the algorithmic space.
Revised: To close this gap, we parameterize the update rule as a neural network and build a Boltzmann-shaped posterior over the algorithmic space.

Original: We apply our Bayesian
Revised: We apply our Bayesian

Original: Learning-to-Optimize (BL2O) framework to optimize test functions, energy functions in protein docking, loss functions in image classiﬁcation and loss functions in data privacy attack.
Revised: Learning-to-Optimize (BL2O) framework to optimize test functions, energy functions in protein docking, loss functions in image classiﬁcation and loss functions in data privacy attack.

Original: The empirical results demonstrate that BL2O outperforms the state-of-the-art methods in both optimization and uncertainty quantiﬁcation, as well as the calibration and out-of-domain detection in classiﬁcation.
Revised: The empirical results demonstrate that BL2O outperforms the state-of-the-art methods in both optimization and uncertainty quantiﬁcation, as well as the calibration and out-of-domain detection in classiﬁcation.

Original: A O PTIMIZER DISTRIBUTION SETTINGS FOR A DAM AND PSO 
Revised: A O PTIMIZER DISTRIBUTION SETTINGS FOR A DAM AND PSO Methods

Original: [Table]
Revised: [Table]

Original: B A NALYTIC FORMS AND M ETA - TRAINING SETS OF T EST FUNCTIONS
Revised: B A NALYTIC FORMS AND M ETA - TRAINING SETS OF T EST FUNCTIONS

Original: [Table]
Revised: [Table]

Original: C S ETTINGS FOR P ROTEIN D OCKING E XPERIMENTS
Revised: C S ETTINGS FOR P ROTEIN D OCKING E XPERIMENTS

Original: We calculate the energy function (objective function f ( x ) ) in a CHARMM 19 force ﬁeld as in (Moal & Bates, 2010).
Revised: We calculate the energy function (objective function f ( x ) ) in a CHARMM 19 force ﬁeld as in (Moal & Bates, 2010).

Original: 25 protein-protein complexes are chosen from the protein docking benchmark set 4.0 (Hwang et al., 2010) as the training set, which is shown in Table 6.
Revised: 25 protein-protein complexes are chosen from the protein docking benchmark set 4.0 (Hwang et al., 2010) as the training set, which is shown in Table 6.

Original: For each target, we choosestarting points (top-5 models from ZDOCK (Pierce et al., 2014)).
Revised: For each target, we choosestarting points (top-5 models from ZDOCK (Pierce et al., 2014)).

Original: In total, our training set includes 125 samples.
Revised: In total, our training set includes 125 samples.

Original: Moreover, we parameterize the search space as R 12 as in BAL (Cao & Shen, 2020).
Revised: Moreover, we parameterize the search space as R 12 as in BAL (Cao & Shen, 2020).

Original: The resulting f ( x ) is fully differentiable in the search space.
Revised: The resulting f ( x ) is fully differentiable in the search space.

Original: We only consider 100 interface atoms due to the computational concern.
Revised: We only consider 100 interface atoms due to the computational concern.

Original: The number of iterations for one training epoch is 600 and in total we have 5000 training epochs.
Revised: The number of iterations for one training epoch is 600 and in total we have 5000 training epochs.

Original: Both BL2O and BAL have 600 iterations during the testing stage.
Revised: Both BL2O and BAL have 600 iterations during the testing stage.

Original: For fair comparison, after optimization, we rescore the BL2O samples for UQ using the scoring function (random forest) in BAL.
Revised: For fair comparison, after optimization, we rescore the BL2O samples for UQ using the scoring function (random forest) in BAL.

Original: [Table]
Revised: [Table]

Original: D P OSTERIOR D ISTRIBUTIONS IN P ROTEIN D OCKING
Revised: D P OSTERIOR D ISTRIBUTIONS IN P ROTEIN D OCKING

Original: || w * , ̂ w || 0.
Revised: || w * , ̂ w || 0.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: D
Revised: D

Original: [Figure]
Revised: [Figure]

Original: n s
Revised: n s

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure] [Table] 
Revised: [Figure] [Table] [Figure]

Original: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: ||
Revised: ||

Original: [Table]
Revised: [Table]

Original: , ̂
Revised: , ̂

Original: [Table]
Revised: [Table]

Original: ||
Revised: ||

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: || w * , ̂ w ||
Revised: || w * , ̂ w ||

Original: [Table]
Revised: [Table]

Original: 0̂ 0̂
Revised: 0̂ 0̂

Original: 1̂ 1̂
Revised: 1̂ 1̂

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: || w
Revised: || w

Original: [Table] [Table]
Revised: [Table] [Table]

Original: w ||
Revised: w ||

Original: [Table]
Revised: [Table]

Original: 1̂
Revised: 1̂

Original: [Table]
Revised: [Table]

Original: || w * , ̂ w || 0.
Revised: || w * , ̂ w || 0.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table]
Revised: [Table]

Original: || w
Revised: || w

Original: [Table]
Revised: [Table]

Original: ̂ w ||
Revised: ̂ w ||

Original: [Table] [Table]
Revised: [Table] [Table]

Original: ||w * , ̂w||
Revised: ||w * , ̂w||

Original: [Figure]
Revised: [Figure]

Original: 0̂ 0̂
Revised: 0̂ 0̂

Original: 1̂
Revised: 1̂

Original: 1̂ 1̂
Revised: 1̂ 1̂

Original: [Figure]
Revised: [Figure]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: E C URRICULUM L EARNING AND C ALIBRATION F IGURE IN I MAGE CLASSIFICATION
Revised: E C URRICULUM L EARNING AND C ALIBRATION F IGURE IN I MAGE CLASSIFICATION

Original: A common issue in learning to optimize for neural network training is that: the optimizer training usually takes hundreds of iterations, while training a neural network usually costs thousands or tens of thousand iterations.
Revised: A common issue in learning to optimize for neural network training is that: the optimizer training usually takes hundreds of iterations, while training a neural network usually costs thousands or tens of thousand iterations.

Original: For instance, for a MNIST training dataset consisting of 50,000 images, training a neural network for 100 epochs corresponds to almost 20,000 iterations with a batch size of 128.
Revised: For instance, for a MNIST training dataset consisting of 50,000 images, training a neural network for 100 epochs corresponds to almost 20,000 iterations with a batch size of 128.

Original: Hundreds of iterations are good for the ﬁrst few epochs during optimizer training, since we would like to only focus on decreasing the loss in the ﬁrst hundreds of iterations.
Revised: Hundreds of iterations are good for the ﬁrst few epochs during optimizer training, since we would like to only focus on decreasing the loss in the ﬁrst hundreds of iterations.

Original: However, as training goes on, we would like to see that the later iterations could also decrease the training loss.
Revised: However, as training goes on, we would like to see that the later iterations could also decrease the training loss.

Original: During this stage, only hundreds of iterations are clearly not enough.
Revised: During this stage, only hundreds of iterations are clearly not enough.

Original: In order to overcome this issue, we bring the idea from (Bengio et al., 2009).
Revised: In order to overcome this issue, we bring the idea from (Bengio et al., 2009).

Original: Speciﬁcally, We set a list for the number of iterations: [100, 200, 500, 1000, 1500, 2000, 2500, 3000] and we gradually increase the number of iterations following the list for every 100 epochs in optimizer training if the optimizee loss is decreasing.
Revised: Speciﬁcally, We set a list for the number of iterations: [100, 200, 500, 1000, 1500, 2000, 2500, 3000] and we gradually increase the number of iterations following the list for every 100 epochs in optimizer training if the optimizee loss is decreasing.

Original: Once the number reaches 3000, it will not change any more until the training ends.
Revised: Once the number reaches 3000, it will not change any more until the training ends.

Original: [Figure] 
Revised: [Figure] F

Revised: A BLATION S TUDY .

Revised: In order to validate various design choices, we perform the ablation study as follows:

Revised: • B1 :

Revised: We use the coordinate-wise gated recurrent unit (GRU) network as the optimizer architecture.

Revised: We train our model directly on the Bayesian loss: Eq 9 without non-Bayesian pre-training.

Revised: • B2 :

Revised: We replace the GRU network with the LSTM network.

Revised: • BL2O : We add the non-Bayesian pre-training stage to ﬁnd a local optimum of interest before training on the Bayesian loss.

Revised: [Figure]

Revised: We test these three models on the Rastrigin test function.

Revised: As shown in Fig 5, B2 (LSTM) has slightly better performance in both optimization and UQ compared to GRU .

Revised: But BL2O has shown superior performance compared to both B1 and B2 in optimization and UQ, which clearly demonstrates the importance of the pre-training for ﬁnding an optimum of interest

Revised: [Figure]