Original: O PEN L OOP H YPERPARAMETER O PTIMIZATION AND D ETERMINANTAL P OINT P ROCESSES
Revised: O PEN L OOP H YPERPARAMETER O PTIMIZATION AND D ETERMINANTAL P OINT P ROCESSES

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Driven by the need for parallelizable hyperparameter optimization methods, this paper studies open loop search methods in the sense that the sequence is predetermined and can be generated before a single conﬁguration is evaluated.
Revised: Driven by the need for parallelizable hyperparameter optimization methods, this paper studies open loop search methods in the sense that the sequence is predetermined and can be generated before a single conﬁguration is evaluated.

Revised: Examples include grid search, uniform random search, low discrepancy sequences, and other sampling distributions.

Original: In particular, we propose the use of k -determinantal point processes in hyperparameter optimization via random search.
Revised: In particular, we propose the use of k -determinantal point processes in hyperparameter optimization via random search.

Original: Compared to conventional  random search where hyperparameter settings are sampled independently, a k -DPP promotes diversity.
Revised: Compared to conventional uniform random search where hyperparameter settings are sampled independently, a k -DPP promotes diversity.

Original: We describe an approach that transforms hyperparameter search spaces for efﬁcient use with a k -DPP.
Revised: We describe an approach that transforms hyperparameter search spaces for efﬁcient use with a k -DPP.

Original: In addition, we introduce a novel Metropolis-Hastings algorithm which can sample from k -DPPs deﬁned over spaces with a mixture of discrete and continuous dimensions.
Revised: In addition, we introduce a novel Metropolis-Hastings algorithm which can sample from kDPPs deﬁned over spaces with a mixture of discrete and continuous dimensions.

Original: Our experiments show signiﬁcant beneﬁts over uniform random search in realistic scenarios with a limited budget for training supervised learners, whether in serial or parallel.
Revised: Our experiments show signiﬁcant beneﬁts over uniform random search in realistic scenarios with a limited budget for training supervised learners, whether in serial or parallel.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Hyperparameter values—regularization strength, model family choices like depth of a neural network or which nonlinear functions to use, procedural elements like dropout rates, stochastic gradient descent step sizes, and data preprocessing choices—can make the difference between a successful application of machine learning and a wasted effort.
Revised: Hyperparameter values—regularization strength, model family choices like depth of a neural network or which nonlinear functions to use, procedural elements like dropout rates, stochastic gradient descent step sizes, and data preprocessing choices—can make the difference between a successful application of machine learning and a wasted effort.

Original: To search among many hyperparameter values requires repeated execution of often-expensive learning algorithms, creating a major obstacle for practitioners and researchers alike.
Revised: To search among many hyperparameter values requires repeated execution of often-expensive learning algorithms, creating a major obstacle for practitioners and researchers alike.

Original: In general, on request/iteration k , a hyperparameter searcher suggests a hyperparameter conﬁguration x k , a worker trains a model using x k , and returns a validation loss of y k computed on a hold out set.
Revised: In general, on request/iteration k , a hyperparameter searcher suggests a hyperparameter conﬁguration x k , a worker trains a model using x k , and returns a validation loss of y k computed on a hold out set.

Original: In this work we say a hyperparameter searcher is open loop if x k depends only on { x i } k − 1 i =1 ; examples include choosing x k uniformly at random (Bergstra et al., 2011a), or x k coming from a low-discrepancy sequence (c.f.,
Revised: In this work we say a hyperparameter searcher is open loop if x k depends only on { x i } k − 1 i =1 ; examples include choosing x k uniformly at random (Bergstra et al., 2011a), or x k coming from a low-discrepancy sequence (c.f.,

Original: Iac ` o (2015)).
Revised: Iac ` o (2015)).

Original: We say a searcher is closed loop if x k depends on both the past conﬁgurations and validation losses { ( x i , y i ) } k − 1 i =1 ; examples include Bayesian optimization (Snoek et al., 2012) and recent reinforcement learning methods (Zoph & Le, 2016).
Revised: We say a searcher is closed loop if x k depends on both the past conﬁgurations and validation losses { ( x i , y i ) } k − 1 i =1 ; examples include Bayesian optimization (Snoek et al., 2012) and recent reinforcement learning methods (Zoph & Le, 2016).

Original: Note that open loop methods can draw an inﬁnite sequence of conﬁgurations before training a single model, whereas closed loop methods rely on validation loss feedback in order to make suggestions.
Revised: Note that open loop methods can draw an inﬁnite sequence of conﬁgurations before training a single model, whereas closed loop methods rely on validation loss feedback in order to make suggestions.

Original: While sophisticated closed loop selection methods have been shown to empirically identify good hyperparameter conﬁgurations faster (i.e., with fewer iterations) than open loop methods like random search, two trends have rekindled interest in embarrassingly parallel open loop methods : 1) modern deep learning models can take days or weeks to train with no signs of efﬁciency breakthroughs, and 2) the rise of cloud resources available to anyone that charge not by the number of machines, but by the number of CPU-hours used so that 10 machines for 100 hours costs the same as 1000 machines for 1 hour.
Revised: While sophisticated closed loop selection methods have been shown to empirically identify good hyperparameter conﬁgurations faster (i.e., with fewer iterations) than open loop methods like random search, two trends have rekindled interest in embarrassingly parallel open loop methods : 1) modern deep learning models can take days or weeks to train with no signs of efﬁciency breakthroughs, and 2) the rise of cloud resources available to anyone that charge not by the number of machines, but by the number of CPU-hours used so that 10 machines for 100 hours costs the same as 1000 machines for 1 hour.

Original: This paper explores the landscape of open loop methods, identifying tradeoffs that are rarely considered, if at all acknowledged.
Revised: This paper explores the landscape of open loop methods, identifying tradeoffs that are rarely considered, if at all acknowledged.

Original: While random search is arguably the most popular open loop method and chooses each x k independently of { x i } k − 1 i =1 , it is by no means the only choice.
Revised: While random search is arguably the most popular open loop method and chooses each x k independently of { x i } k − 1 i =1 , it is by no means the only choice.

Original: In many ways uniform random search is the least interesting of the methods we will discuss because we will advocate for methods where x k depends on { x i } k − 1 i =1 to promote diversity .
Revised: In many ways uniform random search is the least interesting of the methods we will discuss because we will advocate for methods where x k depends on { x i } k − 1 i =1 to promote diversity .

Original: In particular, we will focus on drawing { x i } k i =1 from a k -determinantal point process (DPP) (Kulesza et al., 2012).
Revised: In particular, we will focus on drawing { x i } ki =1 from a k -determinantal point process (DPP) (Kulesza et al., 2012).

Original: DPPs sup- port real, integer, and categorical dimensions—any of which may have a tree structure—and have computationally efﬁcient methods of drawing samples.
Revised: DPPs support real, integer, and categorical dimensions—any of which may have a tree structure—and have computationally efﬁcient methods of drawing samples.

Original: Experimentally, we explore the use of our diversity-promoting open-loop hyperparameter optimization method based on k -DPP random search.
Revised: Experimentally, we explore the use of our diversity-promoting open-loop hyperparameter optimization method based on k -DPP random search.

Original: We ﬁnd that it signiﬁcantly outperforms uniform random search in cases where the hyperparameter values have a large effect on performance.
Revised: We ﬁnd that it signiﬁcantly outperforms uniform random search in cases where the hyperparameter values have a large effect on performance.

Original: Open source implementations of both our hyperparameter optimization algorithm (as an extension to the hyperopt package (Bergstra et al., 2013)) and the MCMC algorithm introduced in Algorithmwill be released upon publication.
Revised: Open source implementations of both our hyperparameter optimization algorithm (as an extension to the hyperopt package (Bergstra et al., 2013)) and the MCMC algorithm introduced in Algorithmwill be released upon publication.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: While this work focuses on open loop methods, the vast majority of recent work on hyperparameter tuning has been on closed loop methods, which we brieﬂy review.
Revised: While this work focuses on open loop methods, the vast majority of recent work on hyperparameter tuning has been on closed loop methods, which we brieﬂy review.

Original: 2. C LOSED L OOP M ETHODS
Revised: 2. C LOSED L OOP M ETHODS

Original: Much attention has been paid to sequential model-based optimization techniques such as Bayesian optimization (Snoek et al., 2012; Bergstra et al., 2011b), which sample hyperparameter spaces adaptively.
Revised: Much attention has been paid to sequential model-based optimization techniques such as Bayesian optimization (Snoek et al., 2012; Bergstra et al., 2011b), which sample hyperparameter spaces adaptively.

Original: These techniques ﬁrst choose a point in the space of hyperparameters, then train and evaluate a model with the hyperparameter values represented by that point, then sample another point based on how well previous point(s) performed.
Revised: These techniques ﬁrst choose a point in the space of hyperparameters, then train and evaluate a model with the hyperparameter values represented by that point, then sample another point based on how well previous point(s) performed.

Original: These methods can become complicated, and while they can lead to improved performance, the differences are frequently small.
Revised: These methods can become complicated, and while they can lead to improved performance, the differences are frequently small.

Original: In addition, it has recently been observed that many Bayesian optimization methods, when run for k iterations, are outperformed by sampling 2 k points uniformly at random (Li et al., 2017).
Revised: In addition, it has recently been observed that many Bayesian optimization methods, when run for k iterations, are outperformed by sampling 2 k points uniformly at random (Li et al., 2017).

Original: Parallelizing Bayesian optimization methods has proven to be nontrivial, and while a number of algorithms exist which sample more than one point at each iteration (Contal et al., 2013;
Revised: Parallelizing Bayesian optimization methods has proven to be nontrivial, and while a number of algorithms exist which sample more than one point at each iteration (Contal et al., 2013;

Original: Desautels et al., 2014; Gonz ´ alez et al., 2016), none can achieve the parallelization that grid search, sampling uniformly, or sampling according to a DPP allow.
Revised: Desautels et al., 2014; Gonz ´ alez et al., 2016), none can achieve the parallelization that grid search, sampling uniformly, or sampling according to a DPP allow.

Original: One recent line of research has examined the use of DPPs for optimizing hyperparameters, in the context of parallelizing Bayesian optimization (Kathuria et al., 2016; Wang et al., 2017).
Revised: One recent line of research has examined the use of DPPs for optimizing hyperparameters, in the context of parallelizing Bayesian optimization (Kathuria et al., 2016; Wang et al., 2017).

Original: At each iteration within one trial of Bayesian optimization, instead of drawing a single new point to evaluate from the posterior, they deﬁne a DPP over a small region of the space and sample a set of diverse points.
Revised: At each iteration within one trial of Bayesian optimization, instead of drawing a single new point to evaluate from the posterior, they deﬁne a DPP over a small region of the space and sample a set of diverse points.

Original: While this can lead to easy parallelization within one iteration of Bayesian optimization, the overall algorithms are still sequential.
Revised: While this can lead to easy parallelization within one iteration of Bayesian optimization, the overall algorithms are still sequential.

Original: Additionally, their approach requires discretizing the hyperparameter space, a drawback which we circumvent.
Revised: Additionally, their approach requires discretizing the hyperparameter space, a drawback which we circumvent.

Original: So-called conﬁguration evaluation methods have been shown to perform well by adaptively allocating resources to different hyperparameter settings (Swersky et al., 2014; Li et al., 2017).
Revised: So-called conﬁguration evaluation methods have been shown to perform well by adaptively allocating resources to different hyperparameter settings (Swersky et al., 2014; Li et al., 2017).

Original: They initially choose a set of hyperparameters to evaluate (often uniformly), then partially train a set of models for these hyperparameters.
Revised: They initially choose a set of hyperparameters to evaluate (often uniformly), then partially train a set of models for these hyperparameters.

Original: After some ﬁxed training budget (e.g. time, or number of training examples observed), they compare the partially trained models against one another and allocate more resources to those which perform best.
Revised: After some ﬁxed training budget (e.g. time, or number of training examples observed), they compare the partially trained models against one another and allocate more resources to those which perform best.

Original: Eventually, these algorithms produce one (or a small number) of fully trained, high-quality models.
Revised: Eventually, these algorithms produce one (or a small number) of fully trained, high-quality models.

Original: In some sense, these approaches are orthogonal to open vs. closed loop methods since both can be applied with these methods.
Revised: In some sense, these approaches are orthogonal to open vs. closed loop methods since both can be applied with these methods.

Original: 2. O PEN L OOP M ETHODS
Revised: 2. O PEN L OOP M ETHODS

Original: As discussed above, recent trends have renewed interest in open loop methods.
Revised: As discussed above, recent trends have renewed interest in open loop methods.

Original: And recently, random search was shown to be competitive with sophisticated closed loop methods for modern hyperparameter optimization tasks like deep networks (Li et al., 2017), inspiring other works to explain the phenomenon (Ahmed et al., 2016).
Revised: And recently, random search was shown to be competitive with sophisticated closed loop methods for modern hyperparameter optimization tasks like deep networks (Li et al., 2017), inspiring other works to explain the phenomenon (Ahmed et al., 2016).

Original: Bergstra & Bengio (2012) offer one of the most comprehensive studies of open loop methods to date, and focus attention on comparing random search and grid search.
Revised: Bergstra & Bengio (2012) offer one of the most comprehensive studies of open loop methods to date, and focus attention on comparing random search and grid search.

Original: A main takeaway of the paper is that uniform random sampling is generally preferred to grid search 1 due to the frequent observation that some hyperparameters have little impact on per- formance, and random search promotes more diversity in the dimensions that matter.
Revised: A main takeaway of the paper is that uniform random sampling is generally preferred to grid search 1 due to the frequent observation that some hyperparameters have little impact on performance, and random search promotes more diversity in the dimensions that matter.

Original: Essentially, if points are drawn uniformly at random in d dimensions but only d (cid:48) <
Revised: Essentially, if points are drawn uniformly at random in d dimensions but only d (cid:48) <

Original: d dimensions are relevant, those same points are uniformly distributed (and just as diverse) in d (cid:48) dimensions.
Revised: d dimensions are relevant, those same points are uniformly distributed (and just as diverse) in d (cid:48) dimensions.

Original: Grid search, on the other hand, distributes conﬁgurations aligned with the axes so if only d (cid:48) <
Revised: Grid search, on the other hand, distributes conﬁgurations aligned with the axes so if only d (cid:48) <

Original: d dimensions are relevant, many conﬁgurations are essentially duplicates.
Revised: d dimensions are relevant, many conﬁgurations are essentially duplicates.

Original: However, grid search does have one favorable property that is clear in just one dimension.
Revised: However, grid search does have one favorable property that is clear in just one dimension.

Original: If k points are distributed on [0 , 1] on a grid, the maximum spacing between points is equal to 1 k − 1 .
Revised: If k points are distributed on [0 , 1] on a grid, the maximum spacing between points is equal to 1 k − 1 .

Original: But if points are uniformly at random drawn on [0 , 1] , the expected largest gap between points scales as 1 √ k .
Revised: But if points are uniformly at random drawn on [0 , 1] , the expected largest gap between points scales as 1 √ k .

Original: If you are unlucky enough to have your minimum located in this largest gap, this difference could be considerable.
Revised: If you are unlucky enough to have your minimum located in this largest gap, this difference could be considerable.

Original: The phenomenon generalizes to higher dimensions but grid search’s advantage does not for the reasons above.
Revised: The phenomenon generalizes to higher dimensions but grid search’s advantage does not for the reasons above.

Original: This is an important concept in numerical integration and one way to quantify this property of a sequence x = ( x 1 , x 2 , . . .
Revised: This is an important concept in numerical integration and one way to quantify this property of a sequence x = ( x 1 , x 2 , . . .

Original: , x k ) is known as star discrepancy:
Revised: , x k ) is known as star discrepancy:

Original: [Equation]
Revised: [Equation]

Original: One can interpret the star discrepancy as a multidimensional version of the Kolmogorov-Smirnov statistic between the sequence x and the uniform measure.
Revised: One can interpret the star discrepancy as a multidimensional version of the Kolmogorov-Smirnov statistic between the sequence x and the uniform measure.

Original: It is well-known that a sequence chosen uniformly at random from [0 , 1] d has an expected star discrepancy of at least (cid:113) 1 k (and is no greater than (cid:113) d log( d ) k ) (Devroye et al., 2013, Corollary 12.5) whereas sequences are known to exist with star discrepancy less than log( k ) d k
Revised: It is well-known that a sequence chosen uniformly at random from [0 , 1] d has an expected star discrepancy of at least (cid:113) 1 k (and is no greater than (cid:113) d log( d ) k ) (Devroye et al., 2013, Corollary 12.5) whereas sequences are known to exist with star discrepancy less than log( k ) d k

Original: Sobol’ (1967), where both bounds depend on absolute constants.
Revised: Sobol’ (1967), where both bounds depend on absolute constants.

Original: These low-discrepancy sequences, as they are known, include the Sobol sequence, which was also given brief mention in (Bergstra & Bengio, 2012) and shown to outperform random search and grid search.
Revised: These low-discrepancy sequences, as they are known, include the Sobol sequence, which was also given brief mention in (Bergstra & Bengio, 2012) and shown to outperform random search and grid search.

Original: We also note that the Sobol sequence is also used as an initialization procedure for some
Revised: We also note that the Sobol sequence is also used as an initialization procedure for some

Original: Bayesian Optimization schemes Snoek et al.
Revised: Bayesian Optimization schemes Snoek et al.

Original: However, the Sobol sequence is only deﬁned for continuous spaces, so for hyperparameter search which involves discrete dimensions it is not appropriate.
Revised: However, the Sobol sequence is only deﬁned for continuous spaces, so for hyperparameter search which involves discrete dimensions it is not appropriate.

Original: The ﬁnal open loop method we study is the DPP, which has been given considerably less attention in the hyperparameter optimization literature.
Revised: The ﬁnal open loop method we study is the DPP, which has been given considerably less attention in the hyperparameter optimization literature.

Original: Comparing the star discrepancy of uniform at random and Sobol, one observes that as d grows large relative to k , Sobol starts to suffer.
Revised: Comparing the star discrepancy of uniform at random and Sobol, one observes that as d grows large relative to k , Sobol starts to suffer.

Original: Indeed, Bardenet & Hardy (2016) notes that the Sobol rate is not even valid until k = Ω(2 d ) which motivates them to study a formulation of a DPP that has a star discrepancy between Sobol and random and holds for all k , small and large.
Revised: Indeed, Bardenet & Hardy (2016) notes that the Sobol rate is not even valid until k = Ω(2 d ) which motivates them to study a formulation of a DPP that has a star discrepancy between Sobol and random and holds for all k , small and large.

Original: They primarily approached this problem from a theoretical perspective, and didn’t include experimental results.
Revised: They primarily approached this problem from a theoretical perspective, and didn’t include experimental results.

Original: Their work, in part, motivates us to look at DPPs as a solution for hyperparameter optimization.
Revised: Their work, in part, motivates us to look at DPPs as a solution for hyperparameter optimization.

Original:  C OMPARISON OF O PEN L OOP M ETHODS
Revised:  C OMPARISON OF O PEN L OOP M ETHODS

Original: Optimization performance–how close a point in our sequence is to the true, ﬁxed minimum–is our goal, not a sequence with low discrepancy.
Revised: Optimization performance–how close a point in our sequence is to the true, ﬁxed minimum–is our goal, not a sequence with low discrepancy.

Original: However, as Bergstra & Bengio (2012) observed, the rare “large gap” that can occur in random sequences without the low discrepancy property can affect optimization performance, on average.
Revised: However, as Bergstra & Bengio (2012) observed, the rare “large gap” that can occur in random sequences without the low discrepancy property can affect optimization performance, on average.

Original: One natural surrogate of average optimization performance is to deﬁne a hyperparameter space on [0 , 1] d and measure the distance from a ﬁxed point, say 12 1 = ( 12 , . .
Revised: One natural surrogate of average optimization performance is to deﬁne a hyperparameter space on [0 , 1] d and measure the distance from a ﬁxed point, say 12 1 = ( 12 , . .

Original: . , 12 ) , to the nearest point in the length k sequence in the Euclidean norm: min i =1 ,...,k || x
Revised: . , 12 ) , to the nearest point in the length k sequence in the Euclidean norm: min i =1 ,...,k || x

Original: i − 1 2 1 || 2 .
Revised: i − 1 2 1 || 2 .

Original: The Euclidean norm (squared) is motivated by a quadratic Taylor series approximation around the minimum of the hypothetical function we wish to minimize.
Revised: The Euclidean norm (squared) is motivated by a quadratic Taylor series approximation around the minimum of the hypothetical function we wish to minimize.

Original: The ﬁrst question we wish to answer is: is low discrepancy a surrogate for optimization performance?
Revised: The ﬁrst question we wish to answer is: is low discrepancy a surrogate for optimization performance?

Original: In the ﬁrst and second columns of Figure 1 we plot the star discrepancy and smallest distance from the center 12 1 , respectively, as a function of the length of the sequence, with each row representing dimensions d=2,3,4, for the
Revised: In the ﬁrst and second columns of Figure 1 we plot the star discrepancy and smallest distance from the center 12 1 , respectively, as a function of the length of the sequence, with each row representing dimensions d=2,3,4, for the

Original: Sobol sequence, uniform at random, and a DPP (see the next section for details).
Revised: Sobol sequence, uniform at random, and a DPP (see the next section for details).

Original: We observe that the Sobol sequence is clearly superior in terms of star discrepancy, with the DPP having a slight edge over Uniform.
Revised: We observe that the Sobol sequence is clearly superior in terms of star discrepancy, with the DPP having a slight edge over Uniform.

Original: However, all methods appear comparable when it comes to distance to the center.
Revised: However, all methods appear comparable when it comes to distance to the center.

Original: [Figure] Acknowledging the fact that practitioners deﬁne the search space themselves more often than not [Figure] we realize that if the search space bounds are too small, the optimal solution often is found on the edge, or in a corner of the hypercube.
Revised: Acknowledging the fact that practitioners deﬁne the search space themselves more often than not, we realize that if the search space bounds are too small, the optimal solution often is found on the edge, or in a corner of the hypercube.

Original: Thus, in some situations it makes sense to bias the sequence towards the edges and the corners, the very opposite of what low discrepancy sequences attempt to do.
Revised: Thus, in some situations it makes sense to bias the sequence towards the edges and the corners, the very opposite of what low discrepancy sequences attempt to do.

Original: While Sobol and uniformly random sequences will not bias themselves towards the corners, a
Revised: While Sobol and uniformly random sequences will not bias themselves towards the corners, a

Original: DPP does.
Revised: DPP does.

Original: This happens because points from a DPP are sampled according to how distant they are from the existing points; this tends to favor points in the corners.
Revised: This happens because points from a DPP are sampled according to how distant they are from the existing points; this tends to favor points in the corners.

Original: This same behavior of sampling in the corners is also very common for Bayesian optimization schemes, which is not surprise due to the known connections between sampling from a DPP and gaussian process (see Section 4.5).
Revised: This same behavior of sampling in the corners is also very common for Bayesian optimization schemes, which is not surprise due to the known connections between sampling from a DPP and gaussian process (see Section 4.5).

Original: In the third column of Figure 1 we plot the distance to the origin which is just an arbitrarily chosen corner of hypercube.
Revised: In the third column of Figure 1 we plot the distance to the origin which is just an arbitrarily chosen corner of hypercube.

Original: As expected, we observe that the DPP tends to outperform uniform at random and Sobol in this metric.
Revised: As expected, we observe that the DPP tends to outperform uniform at random and Sobol in this metric.

Original: In what follows, we study the DPP in more depth and how it performs on real-world hyperparameter tuning problems.
Revised: In what follows, we study the DPP in more depth and how it performs on real-world hyperparameter tuning problems.

Original:  M ETHOD
Revised:  M ETHOD

Original: We begin by reviewing determinantal point processes (DPPs) and k -DPPs.
Revised: We begin by reviewing determinantal point processes (DPPs) and k -DPPs.

Original: Let B be a domain of values from which we would like to sample a ﬁnite subset.
Revised: Let B be a domain of values from which we would like to sample a ﬁnite subset.

Original: (In our use of
Revised: (In our use of

Original: DPPs, this is the set of hyperparameter settings.)
Revised: DPPs, this is the set of hyperparameter settings.)

Original: In general, B could be discrete or continuous; here we assume it is discrete with N values, and we deﬁne Y = { 1 , . . .
Revised: In general, B could be discrete or continuous; here we assume it is discrete with N values, and we deﬁne Y = { 1 , . . .

Original: , N } to be a a set which indexes
Revised: , N } to be a a set which indexes

Original: B (this will be particularly useful in Algorithm 1).
Revised: B (this will be particularly useful in Algorithm 1).

Original: In Section 4.2 we address when B has continuous dimensions.
Revised: In Section 4.2 we address when B has continuous dimensions.

Original: A DPP deﬁnes a probability distribution over 2 Y (all subsets of Y ) with the property that two elements of Y are more (less) likely to both be chosen the more dissimilar (similar) they are.
Revised: A DPP deﬁnes a probability distribution over 2 Y (all subsets of Y ) with the property that two elements of Y are more (less) likely to both be chosen the more dissimilar (similar) they are.

Original: Let random variable Y range over ﬁnite subsets of Y .
Revised: Let random variable Y range over ﬁnite subsets of Y .

Original: There are several ways to deﬁne the parameters of a DPP.
Revised: There are several ways to deﬁne the parameters of a DPP.

Original: We focus on L -ensembles, which deﬁne the probability that a speciﬁc subset is drawn (i.e., P ( Y = A ) for some A ⊂ Y ) as:
Revised: We focus on L -ensembles, which deﬁne the probability that a speciﬁc subset is drawn (i.e., P ( Y = A ) for some A ⊂ Y ) as:

Original: [Equation]
Revised: [Equation]

Original: As shown in Kulesza et al. (2012), this deﬁnition of L admits a decomposition to terms representing the quality and diversity of the elements of Y .
Revised: As shown in Kulesza et al. (2012), this deﬁnition of L admits a decomposition to terms representing the quality and diversity of the elements of Y .

Original: For any y i , y j ∈ Y , let:
Revised: For any y i , y j ∈ Y , let:

Original: [Equation]
Revised: [Equation]

Original: where q i > 0 is the quality of y i , φ
Revised: where q i > 0 is the quality of y i , φ

Original: i ∈ R
Revised: i ∈ R

Original: d is a featurized representation of y i , and K : R d × R d → [0 , 1] is a similarity kernel (e.g. cosine distance).
Revised: d is a featurized representation of y i , and K : R d × R d → [0 , 1] is a similarity kernel (e.g. cosine distance).

Original: (We will discuss how to featurize hyperparameter settings in Section 4.3.)
Revised: (We will discuss how to featurize hyperparameter settings in Section 4.3.)

Original: Here, we ﬁx all q i = 1 ; in future work, closed loop methods might make use of q i to encode evidence about the quality of particular hyperparameter settings to adapt the DPP’s distribution over time.
Revised: Here, we ﬁx all q i = 1 ; in future work, closed loop methods might make use of q i to encode evidence about the quality of particular hyperparameter settings to adapt the DPP’s distribution over time.

Original: 4. S AMPLING FROM A k -DPP
Revised: 4. S AMPLING FROM A k -DPP

Original: DPPs have support over all subsets of Y , including ∅ and Y itself.
Revised: DPPs have support over all subsets of Y , including ∅ and Y itself.

Original: In many practical settings, one may have a ﬁxed budget that allows running the training algorithm k times, so we require precisely k elements of Y for evaluation.
Revised: In many practical settings, one may have a ﬁxed budget that allows running the training algorithm k times, so we require precisely k elements of Y for evaluation.

Original: k -DPPs are distributions over subsets of Y of size k .
Revised: k -DPPs are distributions over subsets of Y of size k .

Original: Thus,
Revised: Thus,

Original: [Equation]
Revised: [Equation]

Original: 4. N EW MCMC A LGORITHM
Revised: 4. N EW MCMC A LGORITHM

Original: Kulesza et al.
Revised: Kulesza et al.

Original: (2012) give an algorithm for sampling exactly from k -DPPs, though it runs in O ( N 3 ) ; a Metropolis-Hastings algorithm presented by Anari et al.
Revised: (2012) give an algorithm for sampling exactly from k -DPPs, though it runs in O ( N 3 ) ; a Metropolis-Hastings algorithm presented by Anari et al.

Original: (2016) is a simple and fast alternative  (included here as Algorithm 1).
Revised: (2016) is a simple and fast alternative [Figure] dimension modestly decrease (included here as Algorithm 1).

Original: Both of these sampling algorithms assume the DPP is deﬁned over a ﬁnite number of items; they are restricted to discrete domains.
Revised: Both of these sampling algorithms assume the DPP is deﬁned over a ﬁnite number of items; they are restricted to discrete domains.

Original: We propose a generalization of the
Revised: We propose a generalization of the

Original: MCMC algorithm which preserves relevant computations while allowing sampling from base sets with discrete dimensions, continuous dimensions, or some continuous and some discrete dimensions (Algorithm 2).
Revised: MCMC algorithm which preserves relevant computations while allowing sampling from base sets with discrete dimensions, continuous dimensions [Figure] or some continuous and some discrete dimensions (Algorithm 2).

Original: To the best of our knowledge, this is the ﬁrst algorithm which allows for sampling from a k -DPP deﬁned over mixed discrete and continuous spaces.
Revised: To the best of our knowledge, this is the ﬁrst algorithm which allows for sampling from a k -DPP deﬁned over mixed discrete and continuous spaces.

Original: Algorithm 1 proceeds as follows: First, initialize a set Y with k indices of L , drawn uniformly.
Revised: Algorithm 1 proceeds as follows: First, initialize a set Y with k indices of L , drawn uniformly.

Original: Then, at each iteration, sample two indices of L (one within and one outside of the set Y ), and with some probability replace the item in Y with the other.
Revised: Then, at each iteration, sample two indices of L (one within and one outside of the set Y ), and with some probability replace the item in Y with the other.

Original: When we have continuous dimensions in the base set, however, we can’t deﬁne the matrix L , so sampling indices from it is not possible.
Revised: When we have continuous dimensions in the base set, however, we can’t deﬁne the matrix L , so sampling indices from it is not possible.

Original: We propose Algorithm 2, which samples points directlyfrom the base set B instead (assuming continuous dimensions are bounded), and computes only the principal minors of L needed for the relevant computations on the ﬂy. Even in the case where the dimensions of B are discrete, Algorithm 2 requires less computation and space than Algorithm 1 (assuming the quality and similarity scores are stored once computed, and retrieved when needed). Previous analyses claimed that Algorithm 1 should be run for O ( N log( N ))
Revised: We propose Algorithm 2, which samples points directly

Original: Input: L , a symmetric, N × N matrix where L i,j = q i q j K ( φ i , φ j ) which deﬁnes a DPP over a ﬁnite base set of items B , and Y = { 1 , . . .
Revised: Input: L , a symmetric, N × N matrix where L i,j = q i q j K ( φ i , φ j ) which deﬁnes a DPP over a ﬁnite base set of items B , and Y = { 1 , . . .

Original: , N } , where Y i indexes a row or column of L
Revised: , N } , where Y i indexes a row or column of L

Original: Output do 3:  v ∈ 
Revised: Output: B Y (the points in B indexed by Y ) 1: Initialize Y to k elements sampled from Y uniformly 2: while not mixed do 3: uniformly sample u ∈ Y , v ∈ Y \ Y 4: set Y (cid:48) =

Original: 4: set Y (cid:48) = Y ∪ { v } \ { u } 5: p ← 12 min  (  det 
Revised: Y ∪ { v } \ { u } 5: p ← 12 min (1 , det ( L Y (cid:48) ) det ( L Y ) ) 6: with probability p : Y = Y (cid:48) 7: Return B Y from the base set B instead (assuming continuous dimensions are bounded), and computes only the principal minors of L needed for the relevant computations on the ﬂy. Input i

Original: L Y
Revised: Output β , a points in B Initialize β k points sampled B uniformly while mixed do 3: uniformly

Original: ) det (
Revised: , v

Original: Y ) ) 6: with probability p :
Revised: B \ β 4: set β (cid:48)

Original: Y = Y (cid:48) 7: Return compute the quality score for each item, q i = Ψ
Revised: = β ∪ { v } \ { u } 5: compute the quality score for each item, q i = Ψ

Original: ( β i ) , ∀ i , and q (cid:48)
Revised: ( β i ) , ∀ i , and q (cid:48)

Original: i =
Revised: i =

Original: Ψ ( β (cid:48) i ) , ∀ i 6: construct L β =
Revised: Ψ ( β (cid:48) i ) , ∀ i 6: construct L β =

Original: [ q i q j K ( Φ ( β i ) , Φ ( β j ))] , ∀ i, j 7: construct L β (cid:48)
Revised: [ q i q j K ( Φ ( β i ) , Φ ( β j ))] , ∀ i, j 7: construct L β (cid:48)

Original: =
Revised: =

Original: [ q (cid:48)
Revised: [ q (cid:48)

Original: i q (cid:48)
Revised: i q (cid:48)

Original: j K ( Φ ( β (cid:48) i ) , Φ ( β (cid:48) j ))] , ∀ i, j 8: p ← 12 min (1 , det ( L β (cid:48) ) det ( L β ) ) 9: with probability p : β = β (cid:48) 10: Return βsteps.
Revised: j K ( Φ ( β (cid:48) i ) , Φ ( β (cid:48) j ))] , ∀ i, j 8: p ← 12 min (1 , det ( L β (cid:48) ) det ( L β ) ) 9: with probability p : β = β (cid:48) 10: Return β

Revised: Even in the case where the dimensions of B are discrete, Algorithm 2 requires less computation and space than Algorithm 1 (assuming the quality and similarity scores are stored once computed, and retrieved when needed).

Revised: Previous analyses claimed that Algorithm 1 should be run for O ( N log( N )) steps.

Original: There are O ( N 2 ) computations required to compute the full matrix L , and at each iteration we will compute at most O ( k ) new elements of L , so even in the worst case we will save space and computation whenever k log( N ) < N .
Revised: There are O ( N 2 ) computations required to compute the full matrix L , and at each iteration we will compute at most O ( k ) new elements of L , so even in the worst case we will save space and computation whenever k log( N ) < N .

Original: In expectation, we will save signiﬁcantly more.
Revised: In expectation, we will save signiﬁcantly more.

Original: 4. C ONSTRUCTING L FOR HYPERPARAMETER OPTIMIZATION
Revised: 4. C ONSTRUCTING L FOR HYPERPARAMETER OPTIMIZATION

Original: The vector φ i will encode y
Revised: The vector φ i will encode y

Original: i (an element of Y ), which in its most general form is an attribute-value mapping assigning values to different hyperparameters.
Revised: i (an element of Y ), which in its most general form is an attribute-value mapping assigning values to different hyperparameters.

Original: Let φ i be a feature vector for y i ∈ Y , a modular encoding of the attribute-value mapping, in which ﬁxed segments of the vector are assigned to each hyperparameter attribute (e.g., the dropout rate, the choice of nonlinearity, etc.).
Revised: Let φ i be a feature vector for y i ∈ Y , a modular encoding of the attribute-value mapping, in which ﬁxed segments of the vector are assigned to each hyperparameter attribute (e.g., the dropout rate, the choice of nonlinearity, etc.).

Original: For a hyperparameter that takes a numerical value in range [ h min , h max ] , we encode value h using one dimension ( j ) of φ and project into the range [0 , 1] :
Revised: For a hyperparameter that takes a numerical value in range [ h min , h max ] , we encode value h using one dimension ( j ) of φ and project into the range [0 , 1] :

Original: [Equation]
Revised: [Equation]

Original: This rescaling prevents hyperparameters with greater dynamic range from dominating the similarity calculations.
Revised: This rescaling prevents hyperparameters with greater dynamic range from dominating the similarity calculations.

Original: A categorical-valued hyperparameter attribute that takes m values is given m elements of r and a one-hot encoding.
Revised: A categorical-valued hyperparameter attribute that takes m values is given m elements of r and a one-hot encoding.

Original: We then compute similarity using an RBF kernel,
Revised: We then compute similarity using an RBF kernel,

Original: K = exp (cid:16) − ||
Revised: K = exp (cid:16) − ||

Original: [Equation]
Revised: [Equation]

Original: ||
Revised: i

Original: [Equation]
Revised: [Equation]

Original: (cid:17) , and hence label our approach k -DPP-RBF.
Revised: || 2 2 σ(cid:17) , and hence label our approach k -DPP-RBF.

Original: Values for σ 2 lead to models with different properties; when σ 2 is small, points that are spread out have little impact, and when σ 2 is large, the increased repulsion between the points encourages them to be as far apart as possible.
Revised: Values for σ 2 lead to models with different properties; when σ 2 is small, points that are spread out have little impact, and when σ 2 is large, the increased repulsion between the points encourages them to be as far apart as possible.

Original: This tradeoff is represented in Figure 1.
Revised: This tradeoff is represented in Figure 1.

Original: 4. T REE - STRUCTURED HYPERPARAMETERS
Revised: 4. T REE - STRUCTURED HYPERPARAMETERS

Original: Many real-world hyperparameter search spaces are tree-structured.
Revised: Many real-world hyperparameter search spaces are tree-structured.

Original: For example, the number of layers in a neural network is a hyperparameter, and each additional layer adds at least one new hyperparameter which ought to be tuned (the number of nodes in that layer).
Revised: For example, the number of layers in a neural network is a hyperparameter, and each additional layer adds at least one new hyperparameter which ought to be tuned (the number of nodes in that layer).

Original: For a binary hyperparameter like whether or not to use regularization, we use a one-hot encoding.
Revised: For a binary hyperparameter like whether or not to use regularization, we use a one-hot encoding.

Original: When this hyperparameter is “on,” we set the associated regularization strength as above, and when it is “off” we set it to zero.
Revised: When this hyperparameter is “on,” we set the associated regularization strength as above, and when it is “off” we set it to zero.

Original: Intuitively, with all other hyperparameter settings equal, this causes the off-setting to be closest to the least strong regularization.
Revised: Intuitively, with all other hyperparameter settings equal, this causes the off-setting to be closest to the least strong regularization.

Original: One can also treat higher-level design decisions as hyperparameters (Komer et al., 2014), such as whether to train a logistic regression classiﬁer, a convolutional neural network, or a recurrent neural network.
Revised: One can also treat higher-level design decisions as hyperparameters (Komer et al., 2014), such as whether to train a logistic regression classiﬁer, a convolutional neural network, or a recurrent neural network.

Original: In this construction, the type of model would be a categorical variable (and thus get a one-hot encoding), and all child hyperparameters for an “off” model setting (such as the convergence tolerance for logistic regression, when training a recurrent neural network) would be set to zero.
Revised: In this construction, the type of model would be a categorical variable (and thus get a one-hot encoding), and all child hyperparameters for an “off” model setting (such as the convergence tolerance for logistic regression, when training a recurrent neural network) would be set to zero.

Original: 4. C ONNECTION TO G AUSSIAN PROCESSES
Revised: 4. C ONNECTION TO G AUSSIAN PROCESSES

Original: Gaussian processes are used widely in hyperparameter optimization algorithms.
Revised: Gaussian processes are used widely in hyperparameter optimization algorithms.

Original: Hennig & Garnett (2016) claim that sampling from a DPP with kernel K is equivalent to sequentially sampling proportional to the posterior variance of a GP deﬁned with covariance kernel K .
Revised: Hennig & Garnett (2016) claim that sampling from a DPP with kernel K is equivalent to sequentially sampling proportional to the posterior variance of a GP deﬁned with covariance kernel K .

Original: Since the entropy of a
Revised: Since the entropy of a

Original: Gaussian is proportional to the log determinant of the covariance matrix, points drawn from a DPP have probability proportional to exp( information gain ) , and the most probable set from the DPP is the set which maximizes the information gain.
Revised: Gaussian is proportional to the log determinant of the covariance matrix, points drawn from a DPP have probability proportional to exp( information gain ) , and the most probable set from the DPP is the set which maximizes the information gain.

Original:  H YPERPARAMETER O PTIMIZATION E XPERIMENTS
Revised:  H YPERPARAMETER O PTIMIZATION E XPERIMENTS

Original: In this section we present our hyperparameter optimization experiments.
Revised: In this section we present our hyperparameter optimization experiments.

Original: We compare k -DPP-RBF, uniform sampling, and a Bayesian optimization algorithm in Section 5.1.
Revised: We compare k -DPP-RBF, uniform sampling, and a Bayesian optimization algorithm in Section 5.1.

Original: We compare samples drawn using Algorithm 1 (which necessitates discretizing the hyperparameter space) and Algorithm 2 against samples drawn uniformly at random in Section 5.2.
Revised: We compare samples drawn using Algorithm 1 (which necessitates discretizing the hyperparameter space) and Algorithm 2 against samples drawn uniformly at random in Section 5.2.

Original: It is worth noting that as k increases, all sampling methods approach the true optimum.
Revised: It is worth noting that as k increases, all sampling methods approach the true optimum.

Original: 5. C ONVOLUTIONAL NEURAL NETWORKS FOR TEXT CLASSIFICATION
Revised: 5. C ONVOLUTIONAL NEURAL NETWORKS FOR TEXT CLASSIFICATION

Original: Our experiments consider a setting where hyperparameters have a large effect on performance: a convolutional neural network for text classiﬁcation (Kim, 2014).
Revised: Our experiments consider a setting where hyperparameters have a large effect on performance: a convolutional neural network for text classiﬁcation (Kim, 2014).

Original: The task is binary sentiment analysis on the Stanford sentiment treebank (Socher et al., 2013).
Revised: The task is binary sentiment analysis on the Stanford sentiment treebank (Socher et al., 2013).

Original: On this balanced dataset, random guessing leads to 50% accuracy.
Revised: On this balanced dataset, random guessing leads to 50% accuracy.

Original: We use the CNN-non-static model from Kim (2014), with word2vec (Mikolov et al., 2013) vectors.
Revised: We use the CNN-non-static model from Kim (2014), with word2vec (Mikolov et al., 2013) vectors.

Original: The model architecture consists of a convolutional layer, a max-over-time pooling layer, then a fully connected layer leading to a softmax.
Revised: The model architecture consists of a convolutional layer, a max-over-time pooling layer, then a fully connected layer leading to a softmax.

Original: We begin with a search over three hyperparameters, assuming a budget of k = 20 repetitions of training the convolutional neural net.
Revised: We begin with a search over three hyperparameters, assuming a budget of k = 20 repetitions of training the convolutional neural net.

Original: L 2 regularization strengths in the range [ e − 5 , e − 1 ] (or no regularization) and dropout rates in [0 . 0 , 0 . 7] are considered.
Revised: L 2 regularization strengths in the range [ e − 5 , e − 1 ] (or no regularization) and dropout rates in [0 . 0 , 0 . 7] are considered.

Original: We consider three increasingly “easy” ranges for the learning rate:
Revised: We consider three increasingly “easy” ranges for the learning rate:

Original: • Hard: [ e − 5 , e 5 ] , where the majority of the range leads to accuracy no better than chance.
Revised: • Hard: [ e − 5 , e 5 ] , where the majority of the range leads to accuracy no better than chance.

Original: • Medium: [ e − 5 , e − 1 ] , where half of the range leads to accuracy no better than chance.
Revised: • Medium: [ e − 5 , e − 1 ] , where half of the range leads to accuracy no better than chance.

Original: • Easy: [ e − 10 , e − 3 ] , where the entire range leads to models that beat chance.
Revised: • Easy: [ e − 10 , e − 3 ] , where the entire range leads to models that beat chance.

Original: Figure 2 shows the accuracy (averaged over 50 runs) of the best model found after exploring 1, 2, ..., k hyperparameter settings.
Revised: Figure 2 shows the accuracy (averaged over 50 runs) of the best model found after exploring 1, 2, ..., k hyperparameter settings.

Original: We see that k -DPP-RBF ﬁnds better models with fewer iterations necessary than the other approaches, especially in the most difﬁcult case.
Revised: We see that k -DPP-RBF ﬁnds better models with fewer iterations necessary than the other approaches, especially in the most difﬁcult case.

Original: Figure 2 compares the sampling methods against a Bayesian optimization technique using a tree-structured Parzen estimator (BOTPE; Bergstra et al., 2011b).
Revised: Figure 2 compares the sampling methods against a Bayesian optimization technique using a tree-structured Parzen estimator (BOTPE; Bergstra et al., 2011b).

Original: This technique evaluates points sequentially, allowing the model to  choose the next point based on how well previous points performed (a closed loop approach).
Revised: This technique evaluates points sequentially, allowing the model to [Figure] choose the next point based on how well previous points performed (a closed loop approach).

Original: It is state-of-the-art on tree-structured search spaces (though its sequential nature limits parallelization).
Revised: It is state-of-the-art on tree-structured search spaces (though its sequential nature limits parallelization).

Original: Surprisingly, we ﬁnd it performs the worst, even though it takes advantage of additional information.
Revised: Surprisingly, we ﬁnd it performs the worst, even though it takes advantage of additional information.

Original: [Figure] We hypothesize that the exploration/exploitation tradeoff in BO-TPE causes it to commit to more local search before exploring the space fully, thus not ﬁnding hard-to-reach global optima.
Revised: We hypothesize that the exploration/exploitation tradeoff in BO-TPE causes it to commit to more local search before exploring the space fully, thus not ﬁnding hard-to-reach global optima.

Original: Note that when considering points sampled uniformly or from a DPP, the order of the k hyperparameter settings in one trial is arbitrary (though this is not the case with BO-TPE as it is an iterative algorithm).
Revised: Note that when considering points sampled uniformly or from a DPP, the order of the k hyperparameter settings in one trial is arbitrary (though this is not the case with BO-TPE as it is an iterative algorithm).

Original: The variance of the k -DPP methods (not shown for clarity) tends to be high in early iterations, simply because the k samples from a k -DPP are likely to be more diverse than those sampled uniformly, but in all cases the variance of the best of the k points is lower than when sampled uniformly.
Revised: The variance of the k -DPP methods (not shown for clarity) tends to be high in early iterations, simply because the k samples from a k -DPP are likely to be more diverse than those sampled uniformly, but in all cases the variance of the best of the k points is lower than when sampled uniformly.

Original: 5. O PTIMIZING WITHIN RANGES KNOWN TO BE GOOD
Revised: O PTIMIZING WITHIN RANGES KNOWN TO BE GOOD

Original: Zhang & Wallace (2015) analyzed the stability of convolutional neural networks for sentence classiﬁcation with respect to a large set of hyperparameters, and found a set of six which they claimed had the largest impact: the number of kernels, the difference in size between the kernels, the size of each kernel, dropout, regularization strength, and the number of ﬁlters.
Revised: Zhang & Wallace (2015) analyzed the stability of convolutional neural networks for sentence classiﬁcation with respect to a large set of hyperparameters, and found a set of six which they claimed had the largest impact: the number of kernels, the difference in size between the kernels, the size of each kernel, dropout, regularization strength, and the number of ﬁlters.

Original: We optimized over their prescribed “Stable” ranges; average accuracies across 50 trials of hyperparameter optimization are shown in Figure 3, across k = 20 iterations, with each dimension discretized to ﬁve values (for the discretized experiments).
Revised: We optimized over their prescribed “Stable” ranges; average accuracies across 50 trials of hyperparameter optimization are shown in Figure 3, across k = 20 iterations, with each dimension discretized to ﬁve values (for the discretized experiments).

Original: For both uniform sampling and sampling using k -DPP-RBF, discretizing the search space hurts performance, thus motivating the use of Algorithm 2.
Revised: For both uniform sampling and sampling using k -DPP-RBF, discretizing the search space hurts performance, thus motivating the use of Algorithm 2.

Original: Additionally, we ﬁnd that even in this case where every value gives reasonable performance, k -DPP-RBF sampling outperforms uniform sampling.
Revised: Additionally, we ﬁnd that even in this case where every value gives reasonable performance, k -DPP-RBF sampling outperforms uniform sampling.

Original: Our experiments reveal that, while the hyperparameters proposed by Zhang & Wallace (2015), can have an effect, the learning rate, which they don’t analyze, is at least as impactful.
Revised: Our experiments reveal that, while the hyperparameters proposed by Zhang & Wallace (2015), can have an effect, the learning rate, which they don’t analyze, is at least as impactful.

Original:  C ONCLUSIONS
Revised:  C ONCLUSIONS

Original: We have explored open loop hyperparameter optimization built on sampling from k -DPPs.
Revised: We have explored open loop hyperparameter optimization built on sampling from k -DPPs.

Original: We described how to construct k -DPPs over hyperparameter search spaces, and showed that sampling from these retains the attractive parallelization capabilities of random search.
Revised: We described how to construct k -DPPs over hyperparameter search spaces, and showed that sampling from these retains the attractive parallelization capabilities of random search.

Original: Our experiments demonstrate that, under a limited computation budget, on a number of realistic hyperparameter optimization problems, these approaches perform better than sampling uniformly at random.
Revised: Our experiments demonstrate that, under a limited computation budget, on a number of realistic hyperparameter optimization problems, these approaches perform better than sampling uniformly at random.

Original: As we increasethe difﬁculty of our hyperparameter optimization problem (i.e., as values which lead to good model
Revised: As we increase

Original: [Figure]
Revised: [Figure]

Original: A v e r a g e b e s t - f o un d m o d e l a cc u r a c y
Revised: A v e r a g e b e s t - f o un d m o d e l a cc u r a c y

Original: [Figure]
Revised: [Figure]

Original:  evaluations become more scarce) the improvement over sampling uniformly at random increases.
Revised: the difﬁculty of our hyperparameter optimization problem (i e., as values which lead to good model evaluations become more scarce) the improvement over sampling uniformly at random increases.

Original: An open-source implementation of our method is available
Revised: An open-source implementation of our method is available