Original: C ONTINUOUS -F IDELITY B AYESIAN O PTIMIZATION WITH K NOWLEDGE G RADIENT
Revised: C ONTINUOUS -F IDELITY B AYESIAN O PTIMIZATION WITH K NOWLEDGE G RADIENT

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: While Bayesian optimization (BO) has achieved great success in optimizing expensive-to-evaluate black-box functions, especially tuning hyperparameters of neural networks, methods such as random search (Li et al., 2016) and multiﬁdelity BO (e.g. Klein et al.
Revised: While Bayesian optimization (BO) has achieved great success in optimizing expensive-to-evaluate black-box functions, especially tuning hyperparameters of neural networks, methods such as random search (Li et al., 2016) and multiﬁdelity BO (e.g. Klein et al.

Original: that exploit cheap approximations, e.g. training on a smaller training data or with fewer iterations, can outperform standard
Revised: that exploit cheap approximations, e.g. training on a smaller training data or with fewer iterations, can outperform standard

Original: BO approaches that use only full-ﬁdelity observations.
Revised: BO approaches that use only full-ﬁdelity observations.

Original: In this paper, we propose a novel Bayesian optimization algorithm, the continuous-ﬁdelity knowledge gradient (cfKG) method, that can be used when ﬁdelity is controlled by one or more continuous settings such as training data size and the number of training iterations.
Revised: In this paper, we propose a novel Bayesian optimization algorithm, the continuous-ﬁdelity knowledge gradient (cfKG) method, that can be used when ﬁdelity is controlled by one or more continuous settings such as training data size and the number of training iterations.

Original: cfKG characterizes the value of the information gained by sampling a point at a given ﬁdelity, choosing to sample at the point and ﬁdelity with the largest value per unit cost.
Revised: cfKG characterizes the value of the information gained by sampling a point at a given ﬁdelity, choosing to sample at the point and ﬁdelity with the largest value per unit cost.

Original: Furthermore, cfKG can be generalized, following Wu et al.
Revised: Furthermore, cfKG can be generalized, following Wu et al.

Original: (2017), to settings where derivatives are available in the optimization process, e.g. large-scale kernel learning, and where more than one point can be evaluated simultaneously.
Revised: (2017), to settings where derivatives are available in the optimization process, e.g. large-scale kernel learning, and where more than one point can be evaluated simultaneously.

Original: Numerical experiments show that cfKG outperforms state-of-art algorithms when optimizing synthetic functions, tuning convolutional neural networks (CNNs) on CIFAR-10 and SVHN, and in large-scale kernel learning.
Revised: Numerical experiments show that cfKG outperforms state-of-art algorithms when optimizing synthetic functions, tuning convolutional neural networks (CNNs) on CIFAR-10 and SVHN, and in large-scale kernel learning.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: In hyperparameter tuning of machine learning models, we seek to ﬁnd a set of hyperparameters x in some set A to minimize the validation error f
Revised: In hyperparameter tuning of machine learning models, we seek to ﬁnd a set of hyperparameters x in some set A to minimize the validation error f

Original: ( x ) , i.e., to solve min x ∈
Revised: ( x ) , i.e., to solve min x ∈

Original: A f ( x ) (1.1)
Revised: A f ( x ) (1.1)

Original: Evaluating f ( x ) can take substantial time and computational power (Bergstra & Bengio, 2012), and may not provide gradient evaluations.
Revised: Evaluating f ( x ) can take substantial time and computational power (Bergstra & Bengio, 2012), and may not provide gradient evaluations.

Original: Thus, machine learning practitioners have turned to Bayesian optimization for solving (1.1) (Snoek et al., 2012) because it tends to ﬁnd good solutions with few function evaluations (Jones et al., 1998).
Revised: Thus, machine learning practitioners have turned to Bayesian optimization for solving (1.1) (Snoek et al., 2012) because it tends to ﬁnd good solutions with few function evaluations (Jones et al., 1998).

Original: As the computational expense of training and testing a modern deep neural network for a single set of hyperparameters has grown as long as days or weeks, it has become natural to seek ways to solve (1.1) more quickly by supplanting some evaluations of f ( x ) with computationally inexpensive lowﬁdelity approximations.
Revised: As the computational expense of training and testing a modern deep neural network for a single set of hyperparameters has grown as long as days or weeks, it has become natural to seek ways to solve (1.1) more quickly by supplanting some evaluations of f ( x ) with computationally inexpensive lowﬁdelity approximations.

Original: Indeed, when training a neural network or most other machine learning models, we can approximate f ( x ) by training on less than the full training data, or using fewer training iterations.
Revised: Indeed, when training a neural network or most other machine learning models, we can approximate f ( x ) by training on less than the full training data, or using fewer training iterations.

Original: Both of these controls on ﬁdelity can be set to achieve either better accuracy or lower computational cost across a range of values reasonably modeled as continuous.
Revised: Both of these controls on ﬁdelity can be set to achieve either better accuracy or lower computational cost across a range of values reasonably modeled as continuous.

Original: In this paper, we consider optimization with evaluations of multiple ﬁdelities and costs where the ﬁdelity is controlled by one or more continuous parameters.
Revised: In this paper, we consider optimization with evaluations of multiple ﬁdelities and costs where the ﬁdelity is controlled by one or more continuous parameters.

Original: We model these evaluations by a realvalued function g ( x, s ) where f ( x ) :
Revised: We model these evaluations by a realvalued function g ( x, s ) where f ( x ) :

Original: =
Revised: =

Original: g ( x, 1 m ) and s ∈ [0 , 1] m denotes the m ﬁdelity-control parameters.
Revised: g ( x, 1 m ) and s ∈ [0 , 1] m denotes the m ﬁdelity-control parameters.

Original: g ( x, s ) can be evaluated, optionally with noise, at a cost that depends on x and s .
Revised: g ( x, s ) can be evaluated, optionally with noise, at a cost that depends on x and s .

Original: In the context of hyperparameter tuning, we may take m = 2 and let g ( x, s 1 , s 2 ) denote the loss on the validation set when training using hyperparameters x with a fraction s 1 of the training data and a fraction s 2 of some maximum allowed number of training iterations.
Revised: In the context of hyperparameter tuning, we may take m = 2 and let g ( x, s 1 , s 2 ) denote the loss on the validation set when training using hyperparameters x with a fraction s 1 of the training data and a fraction s 2 of some maximum allowed number of training iterations.

Original: We may also set m = 1 and let s index either training data or training iterations.
Revised: We may also set m = 1 and let s index either training data or training iterations.

Revised: We assume A is a compact connected uncountable set into which it is easy to project, such as a hyperrectangle.

Original: This problem setting also appears outside of hyperparameter tuning, in any application where the objective is expensive to evaluate and we may observe cheap low-ﬁdelity approximations parame- terized by a continuous vector.
Revised: This problem setting also appears outside of hyperparameter tuning, in any application where the objective is expensive to evaluate and we may observe cheap low-ﬁdelity approximations parameterized by a continuous vector.

Original: For example, when optimizing a system evaluated via a Monte Carlo simulator, we can evaluate a system conﬁguration approximately by running with fewer replications.
Revised: For example, when optimizing a system evaluated via a Monte Carlo simulator, we can evaluate a system conﬁguration approximately by running with fewer replications.

Original: Also, when optimizing an engineering system modeled by a partial differential equation (PDE), we can evaluate a system conﬁguration approximately by solving the PDE using a coarse grid.
Revised: Also, when optimizing an engineering system modeled by a partial differential equation (PDE), we can evaluate a system conﬁguration approximately by solving the PDE using a coarse grid.

Original: Given this problem setting, we use the knowledge gradient approach (Frazier et al., 2009) to design an algorithm to adaptively select the hyperparameter conﬁguration and ﬁdelity to evaluate, to best support solving (1.1).
Revised: Given this problem setting, we use the knowledge gradient approach (Frazier et al., 2009) to design an algorithm to adaptively select the hyperparameter conﬁguration and ﬁdelity to evaluate, to best support solving (1.1).

Original: By generalizing a computational technique based on the envelope theorem ﬁrst developed in Wu et al.
Revised: By generalizing a computational technique based on the envelope theorem ﬁrst developed in Wu et al.

Original: (2017), our algorithm supports parallel function evaluations, and also can take advantage of derivative observations when they are available.
Revised: (2017), our algorithm supports parallel function evaluations, and also can take advantage of derivative observations when they are available.

Original: This algorithm chooses the point or set of points to evaluate next that maximizes the ratio of the value of information from evaluation against its cost.
Revised: This algorithm chooses the point or set of points to evaluate next that maximizes the ratio of the value of information from evaluation against its cost.

Revised: Unlike most existing work on discrete- and continuous-ﬁdelity Bayesian optimization, Our approach considers the impact of our measurement on the future posterior distribution over the full feasible domain, while existing expected-improvement-based approaches consider its impact at only the point evaluated.

Revised: One exception is the entropy-search-based method [10], which also considers the impact over the full posterior.

Revised: Our approach differs from entropy search in that it chooses points to sample to directly minimize expected simple regret, while entropy search seeks to minimize the entropy of the location or value of the global optimizer, indirectly reducing simple regret.

Original: We summarize our contributions as follows.
Revised: We summarize our contributions as follows.

Original: Contributions of this paper:
Revised: of this paper:

Original: • We develop a novel Bayesian Optimization algorithm, the continuous-ﬁdelity knowledge gradient (cfKG) method, which chooses the point and ﬁdelity to sample next that maximizes the ratio of the value of information to its cost; • After ﬁrst developing this algorithm in the sequential derivative-free setting, and inspired by Wu et al.
Revised: • We develop a novel Bayesian Optimization algorithm, the continuous-ﬁdelity knowledge gradient (cfKG) method, which chooses the point and ﬁdelity to sample next that maximizes the ratio of the value of information to its cost; • After ﬁrst developing this algorithm in the sequential derivative-free setting, and inspired by Wu et al.

Original: (2017), we generalize to settings where function evaluations may be performed in parallel, and where we can access possibly noisy and biased gradient information.
Revised: (2017), we generalize to settings where function evaluations may be performed in parallel, and where we can access possibly noisy and biased gradient information.

Original: To the best of our knowledge, this is the ﬁrst multi-ﬁdelity batch Bayesian optimization algorithm, and also the ﬁrst multi-ﬁdelity algorithm that can leverage gradients.

Original: • We show that our algorithm outperforms a number of start-of-art benchmark algorithms when optimizing common synthetic functions, tuning convolutional neural networks on CIFAR-10 and SVHN, and in a large-scale kernel learning example.
Revised: • We show that our algorithm outperforms a number of start-of-art benchmark algorithms when optimizing common synthetic functions, tuning convolutional neural networks on CIFAR-10 and SVHN, and in a large-scale kernel learning example.

Original: The rest of the paper is organized as follows.
Revised: The rest of the paper is organized as follows.

Original: Sect.
Revised: Sect.

Original: 2 reviews related work.
Revised: 2 reviews related work.

Original: Sect.
Revised: Sect.

Original: 3 presents the cfKG method.
Revised: 3 presents the cfKG method.

Original: Sect.
Revised: Sect.

Original: 4 tests cfKG on benchmarks including synthetic functions and hyperparameter tuning for deep learning and kernel learning.
Revised: 4 tests cfKG on benchmarks including synthetic functions and hyperparameter tuning for deep learning and kernel learning.

Original: Sect.
Revised: Sect.

Original: 5 concludes.
Revised: 5 concludes.

Original:  R ELATED WORK
Revised:  R ELATED WORK

Original: Algorithms exploiting inexpensive low-ﬁdelity approximations for hyperparameter tuning have been proposed both within and outside the ﬁeld of Bayesian optimization.
Revised: Algorithms exploiting inexpensive low-ﬁdelity approximations for hyperparameter tuning have been proposed both within and outside the ﬁeld of Bayesian optimization.

Original: Outside of Bayesian optimization, Li et al.
Revised: Outside of Bayesian optimization, Li et al.

Original: (2016) develops an early-stopping method called Hyperband that can outperform traditional Bayesian optimization when tuning hyperparameters within a random search framework by adaptively allocating a single predeﬁned resource that can be taken to be the number of training iterations or the amount of training data.
Revised: (2016) develops an early-stopping method called Hyperband that can outperform traditional Bayesian optimization when tuning hyperparameters within a random search framework by adaptively allocating a single predeﬁned resource that can be taken to be the number of training iterations or the amount of training data.

Original: In contrast to Hyperband we allow more than one ﬁdelity control, support derivative observations, and use one-step optimality in terms of value vs. cost to select the point or points to sample.
Revised: In contrast to Hyperband we allow more than one ﬁdelity control, support derivative observations, and use one-step optimality in terms of value vs. cost to select the point or points to sample.

Original: We compare against Hyperband in our numerical experiments, and show improved performance training a convolutional neural network.
Revised: We compare against Hyperband in our numerical experiments, and show improved performance training a convolutional neural network.

Original: Using a modiﬁed expected improvement (EI) acquisition function, Huang et al.
Revised: Using a modiﬁed expected improvement (EI) acquisition function, Huang et al.

Original: (2006) and Lam et al.
Revised: (2006) and Lam et al.

Original: (2015) develop methods for multi-ﬁdelity settings with a ﬁnite discrete set of low-ﬁdelity approximations.
Revised: (2015) develop methods for multi-ﬁdelity settings with a ﬁnite discrete set of low-ﬁdelity approximations.

Original: We call this the “discrete-ﬁdelity setting” to contrast it with the continuous-ﬁdelity setting we consider here.
Revised: We call this the “discrete-ﬁdelity setting” to contrast it with the continuous-ﬁdelity setting we consider here.

Original: We do not compare against discrete-ﬁdelity methods in our numerical experiments as both Kandasamy et al.
Revised: We do not compare against discrete-ﬁdelity methods in our numerical experiments as both Kandasamy et al.

Original: (2017) and Klein et al.
Revised: (2017) and Klein et al.

Original: (2017) show that continuous-ﬁdelity methods can ﬁnd good solutions in continuous-ﬁdelity problems as much as an order of magnitude faster than discrete-ﬁdelity ones.
Revised: (2017) show that continuous-ﬁdelity methods can ﬁnd good solutions in continuous-ﬁdelity problems as much as an order of magnitude faster than discrete-ﬁdelity ones.

Original: In the speciﬁc context of hyperparameter tuning, Domhan et al. (2015) proposes a simple early stopping criterion combined with SMAC (an EI acquisition function with a random forest based statistical model) in which evaluation of a conﬁguration is stopped if its performance is predicted to be worse than the current best conﬁguration.
Revised: In the speciﬁc context of hyperparameter tuning, Domhan et al. (2015) proposes a simple early stopping criterion combined with SMAC (an EI acquisition function with a random forest based statistical model) in which evaluation of a conﬁguration is stopped if its performance is predicted to be worse than the current best conﬁguration.

Original: Kandasamy et al.
Revised: Kandasamy et al.

Original: (2016) generalizes the upper conﬁdence bound (UCB) criteria to the discreteﬁdelity setting, and further generalizes to continuous ﬁdelities in Kandasamy et al.
Revised: (2016) generalizes the upper conﬁdence bound (UCB) criteria to the discreteﬁdelity setting, and further generalizes to continuous ﬁdelities in Kandasamy et al.

Original: We compare against Kandasamy et al.
Revised: We compare against Kandasamy et al.

Original: (2017) in our numerical experiments, ﬁnding that cfKG provides improved performance when the budget is sufﬁciently large.
Revised: (2017) in our numerical experiments, ﬁnding that cfKG provides improved performance when the budget is sufﬁciently large.

Original: This may be because the UCB criteria was originally designed for minimizing the cumulative regret and not simple regret, with the simple regret bounds resulting from theoretical analysis of cumulative regret being loose enough to be consistent with the empirical performance gap we observe in our experiments.
Revised: This may be because the UCB criteria was originally designed for minimizing the cumulative regret and not simple regret, with the simple regret bounds resulting from theoretical analysis of cumulative regret being loose enough to be consistent with the empirical performance gap we observe in our experiments.

Original: It may also be because both Kandasamy et al.
Revised: It may also be because both Kandasamy et al.

Original: (2016) and Kandasamy et al.
Revised: (2016) and Kandasamy et al.

Original: (2017) use a two-stage process in which the point to evaluate is selected without considering the ﬁdelity, and the ﬁdelity only selecting afterward.
Revised: (2017) use a two-stage process in which the point to evaluate is selected without considering the ﬁdelity, and the ﬁdelity only selecting afterward.

Original: In contrast, cfKG selects the point and ﬁdelity jointly, and considers the impact of ﬁdelity choice on the best point to sample.
Revised: In contrast, cfKG selects the point and ﬁdelity jointly, and considers the impact of ﬁdelity choice on the best point to sample.

Original: Entropy search has also been extended to the discrete-ﬁdelity setting by Swersky et al.
Revised: Entropy search has also been extended to the discrete-ﬁdelity setting by Swersky et al.

Original: Considering training iterations as a one-dimensional continuous ﬁdelity control, Swersky et al.
Revised: Considering training iterations as a one-dimensional continuous ﬁdelity control, Swersky et al.

Original: (2014) generalizes Swersky et al.
Revised: (2014) generalizes Swersky et al.

Original: to this speciﬁc continuous-ﬁdelity setting.
Revised: to this speciﬁc continuous-ﬁdelity setting.

Original: Klein et al.
Revised: Klein et al.

Original: (2017) adapts entropy search to settings where the size of the training data is a one-dimensional continuous ﬁdelity control.
Revised: (2017) adapts entropy search to settings where the size of the training data is a one-dimensional continuous ﬁdelity control.

Original: McLeod et al.
Revised: McLeod et al.

Original: (2017) extends predictive entropy search to settings with one ﬁdelity control and provides a faster way to compute the acquisition function.
Revised: (2017) extends predictive entropy search to settings with one ﬁdelity control and provides a faster way to compute the acquisition function.

Original: Poloczek et al.
Revised: Poloczek et al.

Original: (2017) develops a knowledge gradient method for the discrete-ﬁdelity setting, but does not consider the continuous-ﬁdelity setting.
Revised: (2017) develops a knowledge gradient method for the discrete-ﬁdelity setting, but does not consider the continuous-ﬁdelity setting.

Revised: Our computational techniques are quite different from the ones developed there, as necessitated by our consideration of continuous ﬁdelities.

Revised: As an advantageous byproduct, our computational techniques also avoid the need to discrete A .

Original: This paper proposes the ﬁrst knowledge gradient method for continuous-ﬁdelity settings, and show how to generalize it to the batch and derivative-enabled settings by generalizing the computational technique developed in Wu et al.
Revised: This paper proposes the ﬁrst knowledge gradient method for continuous-ﬁdelity settings, and show how to generalize it to the batch and derivative-enabled settings by generalizing the computational technique developed in Wu et al.

Original: To the best of our knowledge, this is the ﬁrst multi-ﬁdelity batch Bayesian optimization algorithm.
Revised: To the best of our knowledge, this is the ﬁrst multi-ﬁdelity batch Bayesian optimization algorithm.

Original: The relationship of this paper to the existing multi-ﬁdelity
Revised: The relationship of this paper to the existing multi-ﬁdelity

Original: BO literature is summarized in Table.
Revised: BO literature is summarized in Table.

Original: Literature summary
Revised: Literaturesummary

Original: [Table]
Revised: EI / SMAC Entropy Search

Original: UCB
Revised: UCB

Original: Knowledge Gradient
Revised: KnowledgeGradient

Original: Discreteﬁdelity
Revised: [Table]

Original: Huang et al.
Revised: Huang et al.

Original: Lam et al.
Revised: Lam et al.

Original: Swersky et al.
Revised: Swersky et al.

Original: Kandasamyetal.(2016)
Revised: Kandasamyetal.(2016)

Original: Poloczeketal.(2017)
Revised: Poloczeketal.(2017)

Original: Continuousﬁdelity
Revised: Continuousﬁdelity

Original: [Table]
Revised: Domhan et al. Swersky et al. Klein et al. McLeod et al.

Original: Kandasamyetal
Revised: Kandasamy et al.

Original: [Table]

Original: This paper
Revised: This paper

Original: Our method is also related to knowledge gradient methods for single-ﬁdelity BO.
Revised: Our method is also related to knowledge gradient methods for single-ﬁdelity BO.

Original: In particular, the way in which we extend cfKG from the sequential (one evaluation at a time) derivative-free setting to allow batches of points and gradient observations follows work extending single-ﬁdelity knowledge-gradient methods to the batch and derivative settings in Wu & Frazier (2016) and Wu et al.
Revised: In particular, the way in which we extend cfKG from the sequential (one evaluation at a time) derivative-free setting to allow batches of points and gradient observations follows work extending single-ﬁdelity knowledge-gradient methods to the batch and derivative settings in Wu & Frazier (2016) and Wu et al.

Original: We also generalize the envelope-theorem based computational technique developed for the single-ﬁdelity setting in Wu et al.
Revised: We also generalize the envelope-theorem based computational technique developed for the single-ﬁdelity setting in Wu et al.

Original: (2017) to continuous ﬁdelities.
Revised: (2017) to continuous ﬁdelities.

Original: We compare with single-ﬁdelity knowledge gradient methods in our numerical experiments, and show that cfKG is able to levarage inexpensive low-ﬁdelity observations to provide improved performance in both derivative-free and derivative-enabled settings.
Revised: We compare with single-ﬁdelity knowledge gradient methods in our numerical experiments, and show that cfKG is able to levarage inexpensive low-ﬁdelity observations to provide improved performance in both derivative-free and derivative-enabled settings.

Original:  C ONTINUOUS - FIDELITY KNOWLEDGE GRADIENT
Revised:  C ONTINUOUS - FIDELITY KNOWLEDGE GRADIENT

Original: In this section, we propose the continuous-ﬁdelity knowledge gradient (cfKG), a novel Bayesian optimization algorithm that exploits inexpensive low-ﬁdelity approximations.
Revised: In this section, we propose the continuous-ﬁdelity knowledge gradient (cfKG), a novel Bayesian optimization algorithm that exploits inexpensive low-ﬁdelity approximations.

Original: This algorithm, like most Bayesian optimization algorithms, consists of a statistical model and an acquisition function.
Revised: This algorithm, like most Bayesian optimization algorithms, consists of a statistical model and an acquisition function.

Original: cfKG proceeds iteratively by ﬁtting the statistical model (described below in Section 3.1) to all previously sampled (point, ﬁdelty) pairs, and then ﬁnding and sampling the (point, ﬁdelity) pair that maximizes the acquisition function.
Revised: cfKG proceeds iteratively by ﬁtting the statistical model (described below in Section 3.1) to all previously sampled (point, ﬁdelty) pairs, and then ﬁnding and sampling the (point, ﬁdelity) pair that maximizes the acquisition function.

Original: Once the budget is exhausted, at some iteration N , cfKG returns as its ﬁnal solution the point with the lowest estimated g ( x, 1 m ) .
Revised: Once the budget is exhausted, at some iteration N , cfKG returns as its ﬁnal solution the point with the lowest estimated g ( x, 1 m ) .

Original: To describe cfKG in detail, Sect.
Revised: To describe cfKG in detail, Sect.

Original: 3.1 ﬁrst describes Gaussian process regression for modeling both g ( x, s ) and its cost of evaluation.
Revised: 3.1 ﬁrst describes Gaussian process regression for modeling both g ( x, s ) and its cost of evaluation.

Original: This approach is standard, with the novelty arising in cfKG’s acquisition function and how we optimize it.
Revised: This approach is standard, with the novelty arising in cfKG’s acquisition function and how we optimize it.

Original: Then, Sect.
Revised: Then, Sect.

Original: 3.2 presents the cfKG acquisition function, which values sampling a (point, ﬁdelity) pair according to the ratio of the value of the information gained from sampling that point at that ﬁdelity, to the cost of doing so.
Revised: 3.2 presents the cfKG acquisition function, which values sampling a (point, ﬁdelity) pair according to the ratio of the value of the information gained from sampling that point at that ﬁdelity, to the cost of doing so.

Original: Sect.
Revised: Sect.

Original: 3.3 generalizes an envelope-theorem based computational technique developed in Wu et al.
Revised: 3.3 generalizes an envelope-theorem based computational technique developed in Wu et al.

Original: (2017) to efﬁciently maximize this acquisition function.
Revised: (2017) to efﬁciently maximize this acquisition function.

Original: We discuss extensions to the derivative-enabled setting in Sect.
Revised: We discuss extensions to the derivative-enabled setting in Sect.

Original: 3. G AUSSIAN P ROCESSES
Revised: 3. G AUSSIAN P ROCESSES

Original: We put a Gaussian process (GP) prior (Rasmussen & Williams, 2006) on the function g or its logarithm.
Revised: We put a Gaussian process (GP) prior (Rasmussen & Williams, 2006) on the function g or its logarithm.

Original: We describe this procedure placing the prior on g directly, and then discuss below when we recommend instead placing it on ( x, s )
Revised: We describe this procedure placing the prior on g directly, and then discuss below when we recommend instead placing it on ( x, s )

Original: (cid:55)→ log g ( x, s ) .
Revised: (cid:55)→ log g ( x, s ) .

Original: The GP prior is deﬁned by its mean function µ (0) :
Revised: The GP prior is deﬁned by its mean function µ (0) :

Original: A × [0 , 1] m (cid:55)→ R and kernel function K (0) : { A × [0 , 1] m } × { A × [0 , 1] m } (cid:55)→ R .
Revised: A × [0 , 1] m (cid:55)→ R and kernel function K (0) : { A × [0 , 1] m } × { A × [0 , 1] m } (cid:55)→ R .

Original: These mean and kernel functions have hyperparameters, whose inference we discuss below.
Revised: These mean and kernel functions have hyperparameters, whose inference we discuss below.

Original: We assume that evaluations of g ( x, s ) are subject to additive independent normally distributed noise with common variance σ 2 .
Revised: We assume that evaluations of g ( x, s ) are subject to additive independent normally distributed noise with common variance σ 2 .

Original: We treat the parameter σ 2 as a hyperparameter of our model, and also discuss its inference below.
Revised: We treat the parameter σ 2 as a hyperparameter of our model, and also discuss its inference below.

Original: Our assumption of normally distributed noise with constant variance is common in the BO literature (Klein et al., 2017).
Revised: Our assumption of normally distributed noise with constant variance is common in the BO literature (Klein et al., 2017).

Original: The posterior distribution of g after n function evaluations at points z (1: n ) :
Revised: The posterior distribution of g after n function evaluations at points z (1: n ) :

Original: = { ( x (1) , s (1) ) , ( x (2) , s (2) ) , · · · , ( x ( n ) , s ( n ) ) } with observed values y (1: n ) :
Revised: = { ( x (1) , s (1) ) , ( x (2) , s (2) ) , · · · , ( x ( n ) , s ( n ) ) } with observed values y (1: n ) :

Original: = { y (1) , y (2) , · · · , y ( n ) } remains a Gaussian process (Rasmussen & Williams, 2006), and g | z (1: n ) , y (1: n ) ∼ GP ( µ n , K ( n ) ) with µ n and K ( n ) evaluated at a point z
Revised: = { y (1) , y (2) , · · · , y ( n ) } remains a Gaussian process (Rasmussen & Williams, 2006), and g | z (1: n ) , y (1: n ) ∼ GP ( µ n , K ( n ) ) with µ n and K ( n ) evaluated at a point z

Original: = ( x, s ) (or pair of points z , ˜ z
Revised: = ( x, s ) (or pair of points z , ˜ z

Original: = (˜ x, ˜ s ) ) given as follows 
Revised: = (˜ x, ˜ s ) ) given as follows µ ( n ) ( z ) = µ (0) ( z ) + K (0) (cid:16) z, z (1: n ) (cid:17) (cid:16) K (0) ( z (1: n ) , z (1: n ) )

Original: [Equation] − 1 (cid:16) y (1: n ) − µ ( z (1: n ) ) (cid:17) ,
Revised: + σ 2 I (cid:17) − 1 (cid:16) y (1: n ) − µ ( z (1: n ) ) (cid:17) ,

Original: K ( n ) ( z, ˜ z ) = K (0) ( z, ˜ z ) − K (0) (cid:16) z, z (1: n ) (cid:17) (cid:16) K (0) ( z (1: n ) , z (1: n ) )
Revised: K ( n ) ( z, ˜ z ) = K (0) ( z, ˜ z ) − K (0) (cid:16) z, z (1: n ) (cid:17) (cid:16) K (0) ( z (1: n ) , z (1: n ) )

Original: + σ 2 I (cid:17) − 1 K (0) (cid:16) z (1: n ) , ˜ z (cid:17) .
Revised: + σ 2 I (cid:17) − 1 K (0) (cid:16) z (1: n ) , ˜ z (cid:17) .

Original: This statistical approach contains several hyperparameters: the variance σ 2 , and any parameters in the mean and kernel functions.
Revised: This statistical approach contains several hyperparameters: the variance σ 2 , and any parameters in the mean and kernel functions.

Original: We treat these hyperparameters in a Bayesian way as proposed in
Revised: We treat these hyperparameters in a Bayesian way as proposed in

Original: Snoek et al.
Revised: Snoek et al.

Original: When g is the validation error in a hyperparameter optimization problem, we recommend putting a
Revised: When g is the validation error in a hyperparameter optimization problem, we recommend putting a

Original: GP prior on log g ( x, s ) , rather than on g ( x, s ) directly, because (1) g ( x, s ) is nonnegative and will be allowed to be negative after log scaling, better matching the range of values assumed by the GP, and (2) because g ( x, s ) can climb steeply over several orders of magnitude as we move away from the optimal x , making log g ( x, s ) easier to model.
Revised: GP prior on log g ( x, s ) , rather than on g ( x, s ) directly, because (1) g ( x, s ) is nonnegative and will be allowed to be negative after log scaling, better matching the range of values assumed by the GP, and (2) because g ( x, s ) can climb steeply over several orders of magnitude as we move away from the optimal x , making log g ( x, s ) easier to model.

Original: We analogously train a separate GP on the logarithm of the cost of evaluating g ( x, s ) .
Revised: We analogously train a separate GP on the logarithm of the cost of evaluating g ( x, s ) .

Original: 3. T HE CF KG ACQUISITION FUNCTION
Revised: 3. T HE CF KG ACQUISITION FUNCTION

Original: cfKG samples the point and ﬁdelity that jointly maximize an acquisition function, which we deﬁne in this section by adopting the knowledge gradient concept (Frazier et al., 2009) in the continuousﬁdelity setting to value the information gained through one additional sample.
Revised: cfKG samples the point and ﬁdelity that jointly maximize an acquisition function, which we deﬁne in this section by adopting the knowledge gradient concept (Frazier et al., 2009) in the continuousﬁdelity setting to value the information gained through one additional sample.

Original: If we were to stop sampling after n samples, we would select as our solution to (1.1) a point x with minimum estimated validation error µ ( n ) ( x, 1 m ) , and this point would have a conditional expected validation error of min x ∈ A µ ( n ) ( x, 1 m ) under the posterior.
Revised: If we were to stop sampling after n samples, we would select as our solution to (1.1) a point x with minimum estimated validation error µ ( n ) ( x, 1 m ) , and this point would have a conditional expected validation error of min x ∈ A µ ( n ) ( x, 1 m ) under the posterior.

Original: If instead we took an additional sample at x ( n +1) with the ﬁdelity s ( n +1) , then the minimum expected validation error under the resulting posterior would become min x ∈ A µ ( n +1)
Revised: If instead we took an additional sample at x ( n +1) with the ﬁdelity s ( n +1) , then the minimum expected validation error under the resulting posterior would become min x ∈ A µ ( n +1)

Original: ( x, 1 m ) .
Revised: ( x, 1 m ) .

Original: This quantity depends on x ( n +1) and s ( n +1) through the dependence of µ ( n +1)
Revised: This quantity depends on x ( n +1) and s ( n +1) through the dependence of µ ( n +1)

Original: ( x, 1 m ) on the point and ﬁdelity sampled, and is random under the posterior at iteration n because µ ( n +1)
Revised: ( x, 1 m ) on the point and ﬁdelity sampled, and is random under the posterior at iteration n because µ ( n +1)

Original: ( x, 1 m ) depends on the observation y ( n +1) .
Revised: ( x, 1 m ) depends on the observation y ( n +1) .

Original: We discuss this dependence explicitly in Sect.
Revised: We discuss this dependence explicitly in Sect.

Original: The value of the information gained by sampling at x ( n +1) with the ﬁdelity s ( n +1) conditioned on any particular outcome y ( n +1) is thus the difference of these two expected validation errors min x ∈ A µ ( n ) ( x, 1 m )
Revised: The value of the information gained by sampling at x ( n +1) with the ﬁdelity s ( n +1) conditioned on any particular outcome y ( n +1) is thus the difference of these two expected validation errors min x ∈ A µ ( n ) ( x, 1 m )

Original: − min x ∈
Revised: − min x ∈

Original: A µ ( n +1) ( x, 1 m ) .
Revised: A µ ( n +1) ( x, 1 m ) .

Original: We then take the expectation of this difference, over the random outcome y ( n +1) , to obtain the (unconditional) value of the information gained, and take the ratio of this value with the cost of obtaining it to obtain the cfKG acquistion function,cfKG
Revised: We then take the expectation of this difference, over the random outcome y ( n +1) , to obtain the (unconditional) value of the information gained, and take the ratio of this value with the cost of obtaining it to obtain the cfKG acquistion function,

Original: [Equation]
Revised: [Equation]

Original: cost ( n [Equation] [Equation] where cost ( n ) ( x, s ) is the estimated cost of evaluating at x with the ﬁdelity s based on the observations available at iteration n , according to the GP described in Sect.
Revised: where cost ( n ) ( x, s ) is the estimated cost of evaluating at x with the ﬁdelity s based on the observations available at iteration n , according to the GP described in Sect.

Original: 3.1, and E n indicates the expectation taken with respect to the posterior given x (1: n ) , s (1: n ) , y (1: n ) .
Revised: 3.1, and E n indicates the expectation taken with respect to the posterior given x (1: n ) , s (1: n ) , y (1: n ) .

Original: The cfKG algorithm chooses to sample at the point and ﬁdelity that jointly maximize the cfKG acquistion function
Revised: The cfKG algorithm chooses to sample at the point and ﬁdelity that jointly maximize the cfKG acquistion function

Original: [Equation]
Revised: [Equation]

Original: Although this acquisition function considers the expected value of an improvement due to sampling, it differs from expected improvement approaches such as Lam et al.
Revised: Although this acquisition function considers the expected value of an improvement due to sampling, it differs from expected improvement approaches such as Lam et al.

Original: because the point at which an improvement occurs, argmax x ∈ A µ ( n +1) ( x, 1 m ) may differ from the point sampled.
Revised: because the point at which an improvement occurs, argmax x ∈ A µ ( n +1) ( x, 1 m ) may differ from the point sampled.

Original: Moreover, this acquisition function allows joint valuation of both the point x and the ﬁdelity s , while approaches such as Lam et al.
Revised: Moreover, this acquisition function allows joint valuation of both the point x and the ﬁdelity s , while approaches such as Lam et al.

Original: (2015) require valuing a point x assuming it will be evaluated at full ﬁdelity and then choose the ﬁdelity in a second stage.
Revised: (2015) require valuing a point x assuming it will be evaluated at full ﬁdelity and then choose the ﬁdelity in a second stage.

Original: cfKG generalizes naturally to batch settings where we can evaluate multiple (point, ﬁdelity) pairs at once.
Revised: cfKG generalizes naturally to batch settings where we can evaluate multiple (point, ﬁdelity) pairs at once.

Original: We value joint evaluation of q ≥ 1 points x 1: q at ﬁdelities s 1: q , where z 1: q = (( x 1 , s 1 ) , . . .
Revised: We value joint evaluation of q ≥ 1 points x 1: q at ﬁdelities s 1: q , where z 1: q = (( x 1 , s 1 ) , . . .

Original: , ( x q , s q )) , by
Revised: , ( x q , s q )) , by

Original: [Equation]
Revised: [Equation]

Original: We then modify (3.3) by sampling at the batch of points and ﬁdelities that maximize
Revised: We then modify (3.3) by sampling at the batch of points and ﬁdelities that maximize

Original: [Equation]
Revised: [Equation]

Original: Although we have deﬁned the cfKG algorithm’s sampling decision theoretically, (3.3) or (3.5) are challenging optimization problems and naive brute-force approaches are unlikely to produce highquality results with a reasonable amount of computation.
Revised: Although we have deﬁned the cfKG algorithm’s sampling decision theoretically, (3.3) or (3.5) are challenging optimization problems and naive brute-force approaches are unlikely to produce highquality results with a reasonable amount of computation.

Original: Thus, in the next section, we discuss efﬁcient computational methods for solving (3.3) and (3.5).
Revised: Thus, in the next section, we discuss efﬁcient computational methods for solving (3.3) and (3.5).

Original: 3. E NVELOPE - THEOREM - BASED COMPUTATIONAL METHOD
Revised: 3. E NVELOPE - THEOREM - BASED COMPUTATIONAL METHOD

Original: In this section, we describe computational methods for solving (3.3) and (3.5).
Revised: In this section, we describe computational methods for solving (3.3) and (3.5).

Original: We describe our method in the context of (3.5), and observe that (3.3) is a special case.
Revised: We describe our method in the context of (3.5), and observe that (3.3) is a special case.

Original: We generalize a recently proposed envelope-theorem based computational method developed for single-ﬁdelity optimization in Wu et al.
Revised: We generalize a recently proposed envelope-theorem based computational method developed for single-ﬁdelity optimization in Wu et al.

Original: (2017), which is used to provide unbiased estimators of both q-cfKG and its gradient.
Revised: (2017), which is used to provide unbiased estimators of both q-cfKG and its gradient.

Original: We then use stochastic gradient ascent to optimize the q-cfKG acquistion function, optionally with multiple starts.
Revised: We then use stochastic gradient ascent to optimize the q-cfKG acquistion function, optionally with multiple starts.

Original: 3.3. E STIMATING Q - CF KG
Revised: 3.3. E STIMATING Q - CF KG

Original: To support computation, we express µ ( n + q ) ( x, 1 m ) that results from a chosen batch of points and ﬁdelities z ( n +1: n + q ) =
Revised: To support computation, we express µ ( n + q ) ( x, 1 m ) that results from a chosen batch of points and ﬁdelities z ( n +1: n + q ) =

Original: z 1: q as 
Revised: z 1: q as µ

Original: [Equation]
Revised: [Equation]

Revised: n

Revised: [Equation]

Revised: x,

Revised: [Equation]

Revised: n )

Original:  Because y ( n +1: n + q ) − µ ( n ) ( z 1: q ) is normally distributed with zero mean and covariance matrix (cid:0) K ( n ) ( z 1: q , z 1: q ) + σ 2 I (cid:1) with respect to the posterior after n observations, we can rewrite µ ( n + q ) ( x, 1 m ) as
Revised: [Equation] x [Equation] Because y ( n +1: n + q ) − µ ( n ) ( z 1: q ) is normally distributed with zero mean and covariance matrix (cid:0) K ( n ) ( z 1: q , z 1: q ) + σ 2 I (cid:1) with respect to the posterior after n observations, we can rewrite µ ( n + q ) ( x, 1 m ) as

Original: [Equation]
Revised: [Equation]

Original: where W q is a standard q -dimensional normal random vector, and
Revised: where W q is a standard q -dimensional normal random vector, and

Original: [Equation]
Revised: [Equation]

Original: where D ( n ) ( z 1: q ) is the Cholesky factor of the covariance matrix K ( n ) ( z 1: q , z 1: q ) + σ 2 I .
Revised: where D ( n ) ( z 1: q ) is the Cholesky factor of the covariance matrix K ( n ) ( z 1: q , z 1: q ) + σ 2 I .

Original: Thus, to provide an unbiased Monte Carlo estimator of the expectation within (3.4), we may sample
Revised: Thus, to provide an unbiased Monte Carlo estimator of the expectation within (3.4), we may sample

Original: W q , and then calculate min x (cid:48)
Revised: W q , and then calculate min x (cid:48)

Original: ∈
Revised: ∈

Original: A µ ( n + q ) ( x (cid:48) , 1 m ) =
Revised: A µ ( n + q ) ( x (cid:48) , 1 m ) =

Original: min x (cid:48) ∈
Revised: min x (cid:48) ∈

Original: A µ ( n ) ( x, 1 m ) + ˜ σ n ( x, z 1: q ) W q .
Revised: A µ ( n ) ( x, 1 m ) + ˜ σ n ( x, z 1: q ) W q .

Original: To do this optimization, we use a second-order continuous optimization method, where the gradient and
Revised: To do this optimization, we use a second-order continuous optimization method, where the gradient and

Original: Hessian of µ ( n + q ) ( x, 1 m ) with respect to x in (3.6) can be computed by calculating the gradient and Hessian of µ ( n ) ( x, 1 m ) and ˜ σ n ( x, z 1: q ) .
Revised: Hessian of µ ( n + q ) ( x, 1 m ) with respect to x in (3.6) can be computed by calculating the gradient and Hessian of µ ( n ) ( x, 1 m ) and ˜ σ n ( x, z 1: q ) .

Original: We can then compute the q-cfKG acquistion function to arbitrary accuracy by averaging many such independent Monte Carlo estimates.
Revised: We can then compute the q-cfKG acquistion function to arbitrary accuracy by averaging many such independent Monte Carlo estimates.

Original: 3.3. E STIMATING THE GRADIENT OF Q - CF KG
Revised: 3.3. E STIMATING THE GRADIENT OF Q - CF KG

Original: To solve (3.5), we generalize a recently proposed computational method based on the envelope theorem from Wu et al.
Revised: To solve (3.5), we generalize a recently proposed computational method based on the envelope theorem from Wu et al.

Original: (2017) to provide an unbiased estimator of the gradient of the q-cfKG acquisition function, and then use stochastic gradient ascent.
Revised: (2017) to provide an unbiased estimator of the gradient of the q-cfKG acquisition function, and then use stochastic gradient ascent.

Original: Exploiting (3.6), the q-cfKG acquistion function can be expressed as
Revised: Exploiting (3.6), the q-cfKG acquisition function can be expressed as

Original: [Equation]
Revised: [Equation]

Revised: , where

Revised: W q is a standard q -dimensional normal random vector,

Revised: [Equation]

Revised: =

Revised: [Equation]

Revised: D

Revised: [Equation]

Revised: n

Revised: [Equation]

Revised: , and

Revised: D ( n ) ( z 1: q ) is the

Revised: Cholesky factor of the covariance matrix K ( n ) ( z 1: q , z 1: q ) + σ 2 I .

Revised: ∇ q-cfKG ( z 1: q ) can be computed from

Revised: [Equation]

Revised: (cid:1)(cid:3) and ∇ max 1 ≤

Revised: i

Revised: [Equation]

Revised: cost

Revised: [Equation]

Revised: n

Revised: [Equation]

Revised: z

Revised: [Equation]

Revised: , where differentiability of cost ( n ) ( · ) implies max 1 ≤ i ≤ q cost ( n ) ( x i , s i ) is differentiable almost everywhere.

Original: Under sufﬁcient regularity conditions (L’Ecuyer, 1990) that we conjecture hold in most applications to hyperparameter tuning, one can interchange the gradient and expectation operators,
Revised: To compute the ﬁrst term, under sufﬁcient regularity conditions (L’Ecuyer, 1990) that we conjecture hold in most applications to hyperparameter tuning, one can interchange the gradient and expectation operators,

Original: [Equation]
Revised: [Equation]

Original: This technique is called inﬁnitesimal perturbation analysis (IPA) (L’Ecuyer, 1990).
Revised: This technique is called inﬁnitesimal perturbation analysis (IPA) (L’Ecuyer, 1990).

Original: Since multiplication, matrix inversion (when the inverse exists), and Cholesky factorization (Smith, 1995) preserve continuous differentiability,we have ( x, z 1: q ) (cid:55)→ (cid:0) µ ( n ) ( x, 1 m ) + ˜ σ n ( x, z 1: q ) W q (cid:1) is continuously differentiable under mild regularity conditions.
Revised: Since multiplication, matrix inversion (when the inverse exists), and Cholesky factorization (Smith, 1995) preserve continuous differentiability, ( x, z 1: q ) (cid:55)→ (cid:0) µ ( n ) ( x, 1 m ) + ˜ σ n ( x, z 1: q ) W q (cid:1) is continuously differentiable under mild regularity conditions.

Original: When this function is continuously differentiable and A is compact, the envelope theorem (Milgrom & Segal, 2002, Corollary 4) implies
Revised: When this function is continuously differentiable and A is compact, the envelope theorem (Milgrom & Segal, 2002, Corollary 4) implies

Original: [Equation]
Revised: [Equation]

Original: where x ∗ ( W q ) ∈
Revised: where x ∗ ( W q ) ∈

Original: arg
Revised: arg

Original: min x ∈ A (cid:0) µ ( n ) ( x, 1 m )
Revised: min x ∈ A (cid:0) µ ( n ) ( x, 1 m )

Original: + ˜ σ n ( x, z 1: q ) W q (cid:1) .
Revised: + ˜ σ n ( x, z 1: q ) W q (cid:1) .

Original: We can use this unbiased gradient estimator within stochastic gradient ascent (Harold et al., 2003), optionally with multiple starts, to solve the optimization problem (3.5).This is described in Alg. 1.
Revised: We can use this unbiased gradient estimator within stochastic gradient ascent (Harold et al., 2003) to solve the optimization problem (3.5).

Original: 1: for t = 1 to T do 2: for j = 1 to J do 3:

Original: Sample W q,i from a standard multivariate normal distribution.

Original: Solve x ∗ ( W q,i ) ∈

Original: arg min x ∈ A (cid:0) µ ( n ) ( x, 1 m ) + ˜ σ n ( x, z 1: q ) W q,i (cid:1) by Newton’s method using multiple starts.

Original: Calculate ∇ q-cfKG i ( z 1: q ) =

Original: −∇ ˜ σ n ( x ∗ ( W q,i ) , z 1: q ) · W q,i 6: end for 7:

Original: Let 1 J (cid:80)

Original: Ji =1 ∇ q-cfKG i ( z 1: q ) be our estimator of ∇ q-cfKG ( z 1: q ) 8:

Original: Update z 1: q using our gradient estimator and a stochastic gradient ascent iterator.

Original: 9: end for 10: return z 1: q .

Original: 3. E XTENSIONS TO DERIVATIVE - ENABLED SETTINGS
Revised: 3. E XTENSIONS TO DERIVATIVE - ENABLED SETTINGS

Original: Following Wu et al.
Revised: Following Wu et al.

Original: (2017), which developed single-ﬁdelity BO methods for use when gradient information is available, we develop a version of the cfKG algorithm, called derivative-enabled cfKG, that can be used when gradients are available in the continuous-ﬁdelity setting.
Revised: (2017), which developed single-ﬁdelity BO methods for use when gradient information is available, we develop a version of the cfKG algorithm, called derivative-enabled cfKG, that can be used when gradients are available in the continuous-ﬁdelity setting.

Original: This can be used to reduce the number of function evaluations required to ﬁnd a high-quality solution to (1.1).
Revised: This can be used to reduce the number of function evaluations required to ﬁnd a high-quality solution to (1.1).

Original: First, observe that a GP ( µ n , K n ) prior on g ( x, s ) implies a multi-output GP (˜ µ n , ˜ K n ) prior on ( g, ∇ x g ) .
Revised: First, observe that a GP ( µ n , K n ) prior on g ( x, s ) implies a multi-output GP (˜ µ n , ˜ K n ) prior on ( g, ∇ x g ) .

Original: Then, observe that the same reasoning we used to develop the cfKG acquistion function in (3.2) can be used when when we observe gradients to motivate the acquisition function, cf-d-KG ( x, s ) =
Revised: Then, observe that the same reasoning we used to develop the cfKG acquistion function in (3.2) can be used when when we observe gradients to motivate the acquisition function, cf

Original: [Equation]
Revised: [Equation]

Original: where cost ( n ) ( x, s ) is now our estimate after n samples of the cost of evaluating both g and its gradient with respect to x , and ˜ µ ( n ) 1 ( x, s ) is the posterior mean on g ( x, s ) in the multi-output GP.
Revised: where cost ( n ) ( x, s ) is now our estimate after n samples of the cost of evaluating both g and its gradient with respect to x , and ˜ µ ( n ) 1 ( x, s ) is the posterior mean on g ( x, s ) in the multi-output GP.

Original: The conditional expectation is taken with respect to both the observed function value and gradient.
Revised: The conditional expectation is taken with respect to both the observed function value and gradient.

Original: A batch version of the derivative-enabled cfKG acquistion function can be deﬁned analogously.
Revised: A batch version of the derivative-enabled cfKG acquistion function can be deﬁned analogously.

Original: We use techniques similar to those described in Sect.
Revised: We use techniques similar to those described in Sect.

Original: 3.3 to optimize these acquisition functions.
Revised: 3.3 to optimize these acquisition functions.

Original:  N UMERICAL EXPERIMENTS
Revised:  N UMERICAL EXPERIMENTS

Original: In this section, we compare sequential, batch, and derivative-enabled cfKG with several benchmark algorithms on four synthetic functions, tuning convolutional neural networks on CIFARand SVHN, and on large-scale kernel learning.
Revised: In this section, we compare sequential, batch, and derivative-enabled cfKG with several benchmark algorithms on four synthetic functions, tuning convolutional neural networks on CIFARand SVHN, and on large-scale kernel learning.

Original: Our benchmarks include the traditional Bayesian optimization algorithms KG (Wu & Frazier, 2016) and EI (Jones et al., 1998), and the multi-ﬁdelity
Revised: Our benchmarks include the traditional Bayesian optimization algorithms KG (Wu & Frazier, 2016) and EI (Jones et al., 1998), and the multi-ﬁdelity

Original: Bayesian optimization with continuous approximation algorithm (BOCA) (Kandasamy et al., 2017).
Revised: Bayesian optimization with continuous approximation algorithm (BOCA) (Kandasamy et al., 2017).

Original: We also compare with Hyperband (Li et al., 2016) in the CIFRA-10 experiment, and with derivativeenabled KG (Wu et al., 2017) in the kernel-learning experiment.
Revised: We also compare with Hyperband (Li et al., 2016) in the CIFRA-10 experiment, and with derivativeenabled KG (Wu et al., 2017) in the kernel-learning experiment.

Original: We use squared-exponential kernels with constant mean functions and integrate out the GP hyperparameters by sampling M = 10 sets of hyperparameters using the emcee package (Foreman-Mackey et al., 2013).
Revised: We use squared-exponential kernels with constant mean functions and integrate out the GP hyperparameters by sampling M = 10 sets of hyperparameters using the emcee package (Foreman-Mackey et al., 2013).

Original: 4. O PTIMIZING SYNTHETIC FUNCTIONS
Revised: 4. O PTIMIZING SYNTHETIC FUNCTIONS

Original: Inspired by numerical experiments in Kandasamy et al.
Revised: Inspired by numerical experiments in Kandasamy et al.

Original: (2017), we modify four standard synthetic test functions, 2-d Branin, 3-d Rosenbrock, 3-d Hartmann, and 6-d Hartmann, by adding 1 orﬁdelity controls, as described in detail in Appendix.
Revised: (2017), we modify four standard synthetic test functions, 2-d Branin, 3-d Rosenbrock, 3-d Hartmann, and 6-d Hartmann, by adding 1 ﬁdelity control, as described in detail in Appendix.

Original: A.
Revised: A.

Original: We use cost cost ( x, s ) :
Revised: We use cost cost ( x, s ) :

Original: = (cid:81) mi
Revised: = (cid:81) mi

Original: =1 (0 .
Revised: =1 (0 .

Original: + s i ) to mimic the linear cost of training a neural network with more data and longer iterations, and cost ( z 1: q ) :
Revised: + s i ) to mimic the linear cost of training a neural network with more data and longer iterations, and cost ( z 1: q ) :

Original: = max 1 ≤ j ≤ q cost ( z j ) to mimic wall-clock time for training q neural networks in parallel.
Revised: = max 1 ≤ j ≤ q cost ( z j ) to mimic wall-clock time for training q neural networks in parallel.

Original: Fig.
Revised: Fig.

Original: 1 summarizes the results.
Revised: 1 summarizes the results.

Original: cfKG achieves lower simple regret than competitors at the same cost.
Revised: cfKG achieves lower simple regret than competitors at the same cost.

Original: 4. T UNING CONVOLUTIONAL NEURAL NETWORKS ON CIFAR-10 AND SVHN
Revised: 4. T UNING CONVOLUTIONAL NEURAL NETWORKS ON CIFAR-10 AND SVHN

Original: We use sequential and batch cfKG to tune convolution neural networks (CNNs) on CIFAR-10 and
Revised: We use sequential and batch cfKG to tune convolution neural networks (CNNs) on CIFAR-10 and

Original: SVHN.
Revised: SVHN.

Original: Our CNN consists of 3 convolutional blocks and a softmax classiﬁcation layer.
Revised: Our CNN consists of 3 convolutional blocks and a softmax classiﬁcation layer.

Original: Each convolutional block consists of two convolutional layers with the same number of ﬁlters followed by a max-pooling layer.
Revised: Each convolutional block consists of two convolutional layers with the same number of ﬁlters followed by a max-pooling layer.

Original: There is no dropout or batch-normalization layer.
Revised: There is no dropout or batch-normalization layer.

Original: We split the CIFARdataset into 40000 training samples, 10000 validation samples and 10000 test samples.
Revised: We split the CIFARdataset into 40000 training samples, 10000 validation samples and 10000 test samples.

Original: We split the SVHN training dataset into 67235 training samples and 6000 validation samples, and use the standard 26032 test samples.
Revised: We split the SVHN training dataset into 67235 training samples and 6000 validation samples, and use the standard 26032 test samples.

Original: We apply standard data augmentation: horizontal and vertical shifts, and horizontal ﬂips.
Revised: We apply standard data augmentation: horizontal and vertical shifts, and horizontal ﬂips.

Original: We optimize 5 hyperparameters to minimize the classiﬁcation error on the validation set: the learning rate, batch size, and number of ﬁlters in each convolutional block.
Revised: We optimize 5 hyperparameters to minimize the classiﬁcation error on the validation set: the learning rate, batch size, and number of ﬁlters in each convolutional block.

Original: cfKG and q-cfKG use two ﬁdelity controls: the size of the training set and the number of training iterations.
Revised: cfKG and q-cfKG use two ﬁdelity controls: the size of the training set and the number of training iterations.

Original: Hyperband uses the size of the training set as its resource (it can use only one resource or ﬁdelity), using a bracket size of s max = 4 as in Li et al.
Revised: Hyperband uses the size of the training set as its resource (it can use only one resource or ﬁdelity), using a bracket size of s max = 4 as in Li et al.

Original: and the maximum resource allowed by a single conﬁguration set to 40000.
Revised: and the maximum resource allowed by a single conﬁguration set to 40000.

Original: We set the maximum number of training epochs for all algorithms to 50 for CIFAR-10 and 40 for SVHN.
Revised: We set the maximum number of training epochs for all algorithms to 50 for CIFAR-10 and 40 for SVHN.

Original: Fig.
Revised: Fig.

Original: 2 shows the performance of cfKG relative to several  benchmarks.
Revised: 2 shows the performance of cfKG relative to several [Figure] benchmarks.

Original: cfKG successfully exploits the cheap approximations and ﬁnd a good solution much faster than KG and Hyperband.
Revised: cfKG successfully exploits the cheap approximations and ﬁnd a good solution much faster than KG and Hyperband.

Original: When we train using optimized hyperparameters on the full training dataset for 200 epochs, test data classiﬁcation error is ∼ 12% for CIFAR-10 and ∼ 5% for SVHN.
Revised: When we train using optimized hyperparameters on the full training dataset for 200 epochs, test data classiﬁcation error is ∼ 12% for CIFAR-10 and ∼ 5% for SVHN [Figure]

Original: 4. L ARGE - SCALE KERNEL LEARNING
Revised: 4. L ARGE - SCALE KERNEL LEARNING

Original: We use derivative-enabled cfKG (cf-d-KG) in a large-scale kernel learning example obtained by modifying the 1-d demo example for KISS-GP (Wilson & Nickisch, 2015) on the GPML website[Figure] (Rasmussen & Nickisch, 2016).
Revised: We use derivative-enabled cfKG (cf-d-KG) in a large-scale kernel learning example obtained by modifying the 1-d demo example for KISS-GP (Wilson & Nickisch, 2015) on the GPML website (Rasmussen & Nickisch, 2016).

Original: In this example, we optimize 3 hyperparameters (the alpha, the length scale and the variance of the noise) of a GP with an RBF kernel on 1 million training points to maximize the log marginal likelihood.
Revised: In this example, we optimize 3 hyperparameters (the alpha, the length scale and the variance of the noise) of a GP with an RBF kernel on 1 million training points to maximize the log marginal likelihood.

Original: We evaluate both the log marginal likelihood and its gradient using the KISS-GP framework.
Revised: We evaluate both the log marginal likelihood and its gradient using the KISS-GP framework.

Original: We use two ﬁdelity controls: the number of training points and the number of inducing points in the KISS-GP framework.
Revised: We use two ﬁdelity controls: the number of training points and the number of inducing points in the KISS-GP framework.

Original: We set the maximum number of inducing points to m = 1000 .
Revised: We set the maximum number of inducing points to m = 1000 .

Original: We compare cf-d-KG to the derivative-enabled knowledge gradient (d-KG) (Wu et al., 2017) both with batch size q = 8 .
Revised: We compare cf-d-KG to the derivative-enabled knowledge gradient (d-KG) (Wu et al., 2017) both with batch size q = 8 .

Original: Fig.
Revised: Fig.

Original: 2 shows that cf-d-KG successfully utilizes the inexpensive function and gradient evaluations to ﬁnd a good solution more quickly
Revised: 2 shows that cf-d-KG successfully utilizes the inexpensive function and gradient evaluations to ﬁnd a good solution more quickly.

Original: [Figure]
Revised: [Figure]

Original: i f i c a t i o n err o r o nt h e v a li d a t i  o n s
Revised: a g e n e g a t i v e l o g m a r g

Revised: i n a l li k e li h oo d 3d KISSGP

Original: [Figure]
Revised: [Figure]

Original: t
Revised: cf

Original: [Figure]
Revised: [Figure]

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We propose a novel continuous-ﬁdelity BO algorithm, cfKG, which generalizes naturally to batch and derivative settings.
Revised: We propose a novel continuous-ﬁdelity BO algorithm, cfKG, which generalizes naturally to batch and derivative settings.

Original: This algorithm can ﬁnd good solutions to global optimization problems with less cost than state-of-art algorithms in applications including deep learning and kernel learning.
Revised: This algorithm can ﬁnd good solutions to global optimization problems with less cost than state-of-art algorithms in applications including deep learning and kernel learning

Original: A A DDITIONAL EXPERIMENTAL DETAILS
Revised: A A DDITIONAL EXPERIMENTAL DETAILS

Original: Here we deﬁne in detail the synthetic test functions on which we perform numerical experiments described in Sect.
Revised: Here we deﬁne in detail the synthetic test functions on which we perform numerical experiments described in Sect.

Original: The test functions are:
Revised: The test functions are:

Original: augmented
Revised: augmented-Branin

Original: [Equation]
Revised: [Equation]

Original: augmented-Hartmann ( x,
Revised: augmented Hartmann

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: augmented-Rosenbrock 
Revised: augmented-Rosenbrock (

Original: [Equation]
Revised: [Equation]