Increasing the Scope as You Learn: Adaptive Bayesian Optimization in Nested Subspaces
Anonymous Author(s)
AffiliationAddress email
Abstract
Recent advances have extended the scope of Bayesian optimization (BO) toexpensive-to-evaluate black-box functions with dozens of dimensions, aspiring tounlock impactful applications, for example, in the life sciences, neural architecturesearch, and robotics.
However, a closer examination reveals that the state-of-the-artmethods for high-dimensional Bayesian optimization (HDBO) suffer from degrading performance as the number of dimensions increases or even risk failure ifcertain unverifiable assumptions are not met.
This paper proposes BA X US thatleverages a novel family of nested random subspaces to adapt the space it optimizesover to the problem.
This ensures high performance while removing the risk offailure, which we assert via theoretical guarantees.
A comprehensive evaluationdemonstrates that BA X US achieves better results than the state-of-the-art methodsfor a broad set of applications.
Introduction
The optimization of expensive-to-evaluate black-box functions where no derivative information isavailable has found many applications, for example in chemical engineering [10, 26, 29, 54, 56],materials science [22, 27, 28, 30, 49, 59, 63], aerospace engineering [37, 41], hyperparameteroptimization [4, 31, 36, 57], neural architecture search [34, 53], vehicle design [13, 32], hardwaredesign [18, 46], drug discovery [48], robotics [11, 12, 39, 43, 51] and the life sciences [17, 55, 60].
Here increasing the number of dimensions (or parameters) of the optimization problem usually allowsfor better solutions.
For example, by exposing more process parameters of a chemical reaction, weobtain a more granular control of the process; for a design task, we may optimize a larger number ofdesign decisions jointly; in robotics, we gain access to more sophisticated control policies.
A series of breakthroughs has recently pushed the envelope of high-dimensional Bayesian optimization and facilitated a wider adoption in science and engineering. The key challenge for further scalingis the so-called curse of dimensionality.
The complexity of the task of finding an optimum growsexponentially with the number of dimensions [6, 21].
Recently, methods that rely on local ’trustregions’ have gained popularity. They usually achieve good performance for problems with up toa couple of dozens input parameters.
However, we observe that their performance degrades forhigher-dimensional problems.
This is not surprising, given that trust regions have a smaller volumebut still the full dimensionality of the problem.
Other state-of-the-art methods suppose the existenceof a low-dimensional active subspace and enjoy great scalability if they find such a space.
The caveatis that its existence is usually not known for practical applications.
Moreover, the user needs to ‘guess’a good upper bound on its dimensionality to enjoy a good sample efficiency.
In this work, we propose a theoretically-founded approach for high-dimensional Bayesian optimization, BA X US ( B ayesian optimization with A daptive a x is-aligned U niform S ubspaces), that reliablyachieves a high performance on a comprehensive set of applications.
BA X US utilizes a family of nested embedded spaces to increase the dimensionality of the domain that it optimizes over as itcollects more data.
As a byproduct, BA X US can leverage an active subspace, if it exists, withoutrequiring the user to ’guess’ its dimensionality.
BA X US is based on a novel random linear subspaceembedding that enables a more efficient optimization and has strong theoretical guarantees.
We makethe following contributions:
We develop BA X US that reliably achieves excellent solutions on a broad set of high-dimensionaltasks, outperforming the state-of-the-art.2.
A novel family of nested random embeddings that has the following properties: a)
The BA X USembedding has a strictly higher probability of containing an optimum than the H E SBO embeddingproposed by Nayebi et al.
b)
The BA X US embedding is the optimal embedding of thecount-sketch type, as defined in Def.
c) Its probability containing an optimum converges to theone of the H E SBO embedding as the input dimensionality D → ∞ .3.
A comprehensive evaluation on a representative collection of benchmarks that demonstrates thatBA X US outperforms the state-of-the-art methods.
The remainder of this paper is structured as follows.
Section 2 states the problem and discussesrelated work.
Section 3 presents the BA X US algorithm and the corresponding embedding.
Section 4evaluates BA X US on a variety of benchmarks.
Lastly, we discuss our work in Section 5.
Background
The task is to find a minimizer
[Equation]
where X = [ − 1 , +1] D .
The objective function f : X → R is an expensive-to-evaluate black-boxfunction; hence the number of function evaluations needed to find an optimizer is crucial.
Evaluationsmay be subject to observational noise, i.e., f ( x i ) + ε i , with ε i ∼ N (0 , σ 2 ) .
The focus of this workis on scalable high-dimensional Bayesian optimization , where the input dimensionality D is in thehundreds, and the sampling budget may comprise a thousand or more function evaluations.
Linear embeddings.
A successful approach for HDBO is to assume the existence of an activesubspace [16], i.e., there exist a space Z ⊆ R d e , with d e < D and a function g : Z → R , such thatfor all x : g ( T x ) = f
( x ) where T ∈ R d e × D is a projection matrix projecting x onto Z and d e is theeffective dimensionality of the problem.
In practice, both d e and Z are unknown.
REMBO (Random embedding BO) [65] and H E SBO (Hashing-enhanced subspace BO) [47] tryto capture this active subspace by a randomly chosen linear subspace.
Therefore, they generate arandom projection matrix S ⊺ that maps from a d -dimensional subspace Y ∈ R d with d ≪ D (thetarget space ) to X .
We call d the target dimensionality .
For REMBO, each entry in S ⊺ is normallydistributed.
REMBO uses a heuristic to determine a hyperrectangle in Y that it then optimizesover.
Note that the bounded domain may not contain a point that maps to an optimizer of f , arisk aggravated by distortions introduced by the projection.
[5, 7, 8] proposed ideas to mitigatethe issue.
H E SBO’s random projection assigns one target dimension and sign ( ± 1 ) to each inputdimension.
This embedding is inspired by the count-sketch algorithm [14] for estimating frequentitems in data streams.
The sparse projection matrix S ⊺ is, except for the signs, binary, and each rowhas exactly one non-zero entry.
Even though this embedding avoids REMBO’s distortions, it hasa lower probability of containing the optimum [38].
A LEBO [38] uses a Mahalanobis kernel andimposes linear constraints on the acquisition function to avoid projecting outside of X .
Non-linear embeddings.
Several works use autoencoders to learn non-linear spaces for optimization, trading in sample efficiency.
Tripp et al.
[62] change the training objective of a variationalautoencoder (VAE) [35] to make the target space more suitable for optimization.
They give higherweight to better-performing points when training the variational autoencoder (VAE) and show thatthis improves optimization.
Moriconi et al.
[44] incorporate the training of an autoencoder directlyinto the likelihood maximization of a Gaussian process (GP) surrogate.
The computational cost iscubic in the number of samples and the input dimension.
Lu et al.
[40] and Maus et al.
[42] useautoencoders to learn embeddings of highly structured input spaces such as kernels or molecules.
Other approaches include partial least squares [9] or sliced inverse regression [15].
High-dimensional BO in the input space.
One method to make HDBO in the input space feasibleis trust regions (TRs) [21, 50, 52, 68].
The T U RBO algorithm [21] optimizes over bounded TRsinstead of the global space, adapting their side lengths and the center points during the optimizationprocess.
By restricting function evaluations to trust regions, T U RBO addressed the problem of overexploration; see [21] for details.
Note that the TRs have full input dimensionality, which may impact
T U RBO’s ability to scale to very large dimensions.
Nonetheless, T U RBO set a new state-of-the-artby scaling to dozens on input dimensions and thousands of function evaluations.
Wan et al.
[64]extended the idea of TRs to categorical and mixed spaces by using the Hamming distance to define the
TR boundaries.
SAASBO [19] uses sparse priors on the GP length scales which seems particularlyvaluable if the active subspace is axis-aligned.
Indeed, SAASBO can outperform T U RBO on certainbenchmarks [19].
The cost of inference scales cubically with the number of function evaluations;thus, SAASBO is not expected to scale beyond small sampling budgets, which is confirmed by ourexperiments.
Another line of research relies on the assumption that the input space has an additivestructure [23, 33, 45, 66].
Additive GPs rely on computationally expensive sampling methods tolearn a decomposition of the input variables, which limits the scalability of such methods to problemsof moderate dimensionalities and sampling budgets [21, 47].
In Sect.
4 we evaluate the performances of T U RBO, SAASBO, A LEBO , and H E SBO.
Moreover,we study the popular CMA-ES [24] and random search [3].
The BA X US algorithm
Wang et al.
[65] showed that the REMBO embedding contains the optimum in the target space withprobability 1 if d ≥ d e and there are no bounds on the target and input spaces, i.e., Y = R d and
X = R D .
For d < d e , it is generally impossible to represent an optimum in X for arbitrary f as S ⊺projects to a d -dimensional subspace in X .
We call the probability of a target space to contain theoptimum the success probability .
For d ≥ d e , there is a positive success probability that increaseswith d [38, 65].
The main problem is to set d as small as possible to avoid the detrimental effects ofthe curse of dimensionality but as large as necessary to achieve a high probability of Y to contain anoptimum.
In reality, the active subspace and its dimensionality are usually unknown.
The performance ofmethods, such as REMBO [65], H E SBO [47], and A LEBO [38] depends on choosing d such that thesuccess probability is high.
Therefore, they implicitly rely on guessing the effective dimensionalityd e appropriately.
We argue that choosing the target dimensionality is problematic in many practicalapplications.
If chosen too small, the subspace cannot represent f sufficiently well.
If it is chosen toolarge, the curse of dimensionality slows down the optimization.
The proposed algorithm, BA X US, operates on target spaces of increasing dimensionality whilepreserving previous observations.
Let d init be the initial target dimensionality and m the totalevaluation budget.
BA X US starts with a d init -dimensional
embedding that is increased over timeuntil, after m D ≤ m evaluations, it roughly reaches the input dimensionality D .
With this strategy, wecan leverage the efficiency of BO in low-dimensional spaces while guaranteeing to find an optimumin the limit.
Increasing the target dimensionality is enabled by a novel embedding, which lets us carryover observations from previous, lower-dimensional target spaces into more high-dimensional targetspaces.
We further use a TR-based approach based on Eriksson et al.
[21] to carry out optimizationfor high target dimensions effectively.
BA X US uses a GP surrogate [67] to model the function inthe target space.
Algorithm 1 gives the pseudocode for BA X US.
In Appendix D, we prove globalconvergence for BA X US.
We will now present the different components in detail.
3.1 The Axis-aligned Uniform Subspace Embedding
The BA X US embedding uses a sparse projection matrix to map from Y to X .
The number ofnon-zero entries in this matrix is equal to the input dimensionality D .
Another embedding with thisproperty is the H E SBO embedding [47].
Given the D and a target dimensionality d , H E SBO samplesa target dimension ∈ { 1 , . . .
, d } and a sign ∈ {± 1 } for each input dimension uniformly at random.
Conversely, each target dimension has a set of signed contributing input dimensions.
We call the setof contributing input dimensions to a target dimension a bin .
These relations implicitly define theembedding matrix S ∈ { 0 , ± 1 } d × D , where each column has exactly one non-zero entry [14].
In the H E SBO embedding, the number of contributing input dimensions varies between 0 and D .
S ⊺ = − +   +  − −   
Input space
Before Splitting
Target space - 0.
+ 0. + 0.
xxxxxyysplit 0.
Input space
After Splitting
Target space - 0.
+ 0. + 0.
xxxxxyyy0.
c opy
The interpretation of contributing input dimensions allows for an intuitive way to increase theembedding, which is shown in Figure 1.
We update the embedding matrix such that contributinginput dimensions of the target dimension are re-assigned to the current bin and b new bins.
We thensay that we split the corresponding target dimension.
Importantly, this type of embedding allows forretaining observations (see Figure 1) without any distortions.
Assume, for example, that y i is thedimension to be split.
The contributing input dimensions are re-assigned to y i and two new targetdimensions y j and y k (here, b = 2 ) ; the observations can be retained by setting the value of thecoordinate y i to the coordinates y j and y k .
The observations remain undistorted in the input spacebecause an input dimension re-assigned to a new target dimension still has the same value.
By doingso, the observations from the previous target dimension get carried over to the new target dimension.
Algorithm 2 describes the procedure in detail.
In the BA X US embedding, we force each bin of a target dimension to have roughly the samenumber of contributing input dimensions; the bin sizes differ by at most one.
First, we create arandom permutation of the input dimensions 1 , . . .
, D .
The list of input dimensions is split into E min( d, D ) individual bins.
If d does not divide D , not all bins can have the same size.
We split the permutation of input dimensions such that the i -th bin has size ⌈ D/d ⌉ , if i + d ⌊ D/d ⌋ ≤ D ,and ⌊ D/d ⌋ otherwise.
The first bins have one additional element with this construction if d doesnot divide D .
We further randomly assign a sign to each input dimension.
The sign of the inputdimensions and their assignment to target dimensions then implicitly define S ⊺ (see Figure 1).
Wenow show that target spaces of the BA X US embedding have a higher success probability than for the H E SBO embedding. We establish the following two definitions.
Definition 1 ( Count-sketch type embedding ) .
We say that a matrix S ∈ { 0 , ± 1 } D × d is of thecount-sketch type if and only if each column in S has exactly one non-zero entry.
Definition 2 ( Success of a count-sketch type embedding ) .
A success of count-sketch type embedding is the event Y ∗ =
“For any function f : X → R with an axis-aligned active subspace thatattains minima in X , there exists a y ∗ in Y such that S ⊺ y ∗ ∈ arg min x ∈X f ( x ) ”, where S ⊺ mapsfrom the target space Y to the input space X .
We refer to Definition 4 in Appendix C for a formal definition of a function with an axis-aligned activesubspace.
In Lemma 1 we give the success probability of the BA X US embedding.
All proofs havebeen deferred to Appendix C. Note that other than in the count-sketch algorithm [14], our hashingfunction is not pairwise independent. However, this does not influence our theoretical analysis.
Lemma 1 ( Success probability of the BA X US embedding ) .
Let D be the input dimensionality andd ≥ d e the dimensionality of the embedding.
Let β small = (cid:4) D d (cid:5) and β large =
(cid:6) D d (cid:7) be the small andlarge bin sizes.
Then, the probability of Y ∗ (see Definition 2) with the BA X US embedding is
[Equation]
Figure 2 shows the success probabilities of the BA X US and H E SBO embeddings for three differentsettings of D .
The success probability of the H E SBO embedding is given by p H ( Y ∗ ; d, d e ) =
d ! / (( d − d e )!
d d e ) [38].
It is independent of D but is shown for varying d -ranges on the x -axis,therefore the visual discrepancy.
Except for when D → ∞ , the H E SBO embedding always has anon-zero failure probability, i.e., of not containing a given optimum.
The BA X US embedding ensuresthat the success probability is one whenever d ≥ D .
Discontinuities in the curve of the BA X USembedding occur due to the unequal bin sizes in the BA X US embedding’s success probability.
Thedifference between the two embeddings in Figure 2 is particularly large when d e is high.
For small d e ,both embeddings have a high success probability for moderate d .
If d e is high, for example d e = 20 , H E SBO requires d = 1000 to reach a success probability of approximately 0 .
8 , while the BA X USembedding has a success probability of 1 as soon as d = D .
For finite D , the success probability ofthe H E SBO embedding is lower.
Corollary 1.
For D → ∞ , the success probability of the BA X US embedding is
[Equation]
and hence matches H E SBO ’s success probability
p H ( Y ∗ ; d, d e ) .
We show that the BA X US embedding is optimal among count-sketch type embeddings.
Corollary 2.
With the same input, target, and effective dimensionalities ( D , d , and d e ), no countsketch type embedding has a higher success probability than the BA X US embedding.
3.2 Trust-region approach
Similar to T U RBO [21], BA X US operates in TRs.
TRs are hyper-rectangles in the input space; theirshape is defined by their base side length L and the GP length scales.
A TR always has volume L ,but the side length for each dimension is proportional to the corresponding length scale of the GPkernel fitted to the data.
The idea of this construction is that short GP length scales signify importantdimensions and reduce the side length of the hyper-rectangle in that dimension.
A TR is shrunkwhen T U RBO fails τ fail consecutive times to make progress, i.e., to find a better solution.
If thealgorithm consecutively makes progress for τ success = 3 times, it expands the TR.
It restarts whenthe base side length L of the current TR falls below a threshold L min :
= 2 − 7 .
In that case, it discardsall observations for the TR and initializes a new TR on new samples.
TRs enable T U RBO to focuson regions of the space that are close to the incumbent, i.e., the currently best solution from thealgorithm’s perspective.
To choose the next evaluation point, T U RBO minimizes posterior samplesof the GP on a set of candidates drawn from the TR (Thompson sampling [61]).
TRs are an essential component of BA X US as the target dimensionality grows exponentially.
We usethe same hyperparameter settings as T U RBO [21] with the following exceptions: First, we change thecriterion for when to restart a TR.
Instead of restarting a TR when it becomes too small, we increasethe target dimensionality by splitting each target dimension into several new bins unless BA X US hadalready reached the input dimensionality and we reset the TR base side length to the initial value. By resetting the base side length, we also avoid convergence to a particular local minimum as the TR covers a large regions of the space again.
T U RBO solves this problem by allowing for multipleparallel TRs.
Secondly, we change the fail tolerance τ fail , such that BA X US can roughly reach theinput dimensionality in a fixed number of evaluations.
The method for this is described in Section 3.4.
We use the same algorithmic setup as T U RBO with the aforementioned exceptions.
3.3 Splitting strategy
Starting in a low-dimensional embedded space, BA X US successively grows the target dimensionalityto increase the probability of containing the optimum.
By Corollary 2, it is optimal to keep thenumber of contributing input dimensions in the target bins as equal as possible.
At each splitting point, BA X US re-assigns the contributing input dimension of each target dimension to. Hence, the targetdimensionality grows exponentially.
The number of splits required to reach some input dimensionality
D is logarithmic in D .
Compared to the constant growth of the target dimensionality, BA X US has alarger evaluation budget in each split and is less dependent on the value of D .
The algorithm starts intarget dimensionality d init .
After k splits, the target dimensionality is d k = d init ( b + 1) k .
3.4 Controlling the fail tolerance
As the dimensionality grows, the GP needs more data to model the objective function accurately.
Wepartition the evaluation budget into chunks of increasing sizes such that the sizes are proportional tothe target dimensionalities.
Each target dimensionality then has an individual evaluation budget.
Ourstrategy is to control the budget that BA X US has in each target dimensionality by the fail tolerance.
We choose this indirect approach as we do not want to force BA X US to increase the current targetdimensionality if it makes considerable progress.
The rule we propose is based on the fail toleranceτ fail required for the increasing embedding to reach the input dimensionality D , under the assumptionthat BA X US always fails in making progress.
We calculate the number of splits n required to reach
D by
[Equation]
with ⌊·⌉ indicating rounding to the nearest integer.
We assign each split i a split budget m si that isproportional to d i , such that (cid:80) ni =1 m si = m D , where m D is the evaluation budget until D is reached:m si = (cid:106) − b · m D · d i d init (1 − ( b +1) n +1 ) (cid:109) .
With that budget, we set the raw adjusted fail tolerance in the i -th splitas ˜ τ i fail =
(cid:106) m si k (cid:107) , with k = (cid:106) log 12 L min L init (cid:107) being the number of TR shrinkages until the TR is restarted.
We aim to reduce the fail tolerance to enable BA X US to reach higher target dimensions whennecessary.
We are not interested in having even higher fail tolerances than the T U RBO algorithm, which could occur with this rule.
Therefore, we set the fail tolerance by taking theminimum of the raw adjusted fail tolerance and the T U RBO fail tolerances for single batches,τ i fail = max (cid:0) 1 , min (cid:0) ˜
τ i fail , d i (cid:1)(cid:1) , where d i is the target dimension in the i -th split.
Setting the initial target dimension.
If the target dimensionality is fixed, increasing the targetdimensionality at an exponential rate can lead to cases where none of the target dimensionalities d kare close to the input dimensionality D .
For example, for a 600-dimensional problem, with d init = 1 ,and b = 3 , the difference between d k and D is at least 344.
Since the final target dimensionalityof BA X US is at most D , such cases are in disagreement with the exponential growth of the targetdimensionality: Either the final target dimensionality is much less than the input dimensionality, orthe last split adds only a few new dimensions.
In the first case, we potentially reduce the successprobability.
In the latter case, we assign a too large budget to the final split.
Therefore, we set theinitial target dimensionality such that the final target dimensionality is as close to D as possible:
[Equation]
The optimal n is given by Eq.
Experimental evaluation
In this section, we evaluate the performance of BA X US on a 388D hyperparameter optimization task,124D design problem, and a collection of tasks that exhibit an active subspace.
The code is availableat hiddenforanonymization .
The experimental setup.
We benchmark against T U RBO [21] with one and five trust regions, SAASBO [19], A LEBO [38], random search [3], CMA-ES [24], and H E SBO [47], using theimplementations provided by the respective authors with their settings, unless stated otherwise.
For CMA-ES, we use the P Y C MA [25] implementation.
For H E SBO and A LEBO , we use the A Ximplementation [1]. To show the effect of different choices of d , we run H E SBO and A LEBO withd = 10 and d = 20 .
We observed that A LEBO and SAASBO are constrained by their high runtimeand memory consumption.
The available hardware allowed up to 100 evaluations for SAASBO, 500evaluations for A LEBO .
Larger sampling budgets or higher target dimensions for A LEBO resulted inout-of-memory errors.
We point out that limited scalability was expected for these methods, whereasthe other methods scaled to considerably larger budgets, as required for scalable BO.
We initializeeach optimizer with ten initial samples and BA X US with b = 3 and m D = 1000 and run 20 repeatedtrials.
Plots show the mean performance with one standard error.
The benchmarks.
We evaluate the selected algorithms on six benchmarks that differ considerablyin their characteristics.
Following [65], we augment the B RANIN 2 and H ARTMANN 6 functionswith additional dummy dimensions that have no influence on the function value.
We use the 388D
S VM benchmark and the 124D soft-constraint version of the M OPTA 08 benchmark proposed in [19].
We set a budget of 1000 evaluations for M OPTA 08, B RANIN 2, and H ARTMANN 6 and of 2000evaluations for the other benchmarks.
Moreover, we stopped for B RANIN 2 and H ARTMANN 6 whenthe simple regret dropped below .
We also tested the algorithms on the 300D L ASSO -H IGHand the 1000D L ASSO -H
ARD benchmarks from L ASSO B ENCH [55].
These benchmarks have aneffective dimensionality of 5% of the input dimensionality, i.e., the L ASSO -H IGH and L ASSOH ARD benchmarks have 15 and 50 effective dimensions, respectively.
To study the robustness toobservational noise, we tested on noisy variants of L ASSO -H ARD and L ASSO -H IGH .
4.1 Experimental results
We begin with the six noise-free benchmarks.
Fig.
3 summarizes the performances.
On M OPTA 08,a
124D vehicle design problem
, SAASBO makes slightly faster initial progress than BA X US.
Wesuspect that this benchmark has high effective dimensionality, such that BA X US first needs to adaptthe target dimensionality to make further progress.
On the 388D S VM benchmark, BA X US adapts tothe appropriate target dimensionality where it can reach good function values faster than T U RBO and CMA-ES.
For this benchmark, Eriksson and Jankowiak [19] reported that SAASBO learned threeeffective dimensions.
Yet, the fact that A LEBO and H E SBO seem to stagnate after a few hundredevaluations, while BA X US, T U RBO, and CMA-ES find better solutions, indicates that optimizingmore of the 385 kernel length scales of the S VM benchmark allows for better solutions.
On the 500D H ARTMANN 6, SAASBO performs best, closely followed by BA X US.
A LEBO and H E SBO arecompetitive initially but converge to suboptimal solutions.
H E SBO, BA X US, SAASBO, A LEBO allfind excellent solutions on the B RANIN benchmark, with the latter algorithms converging faster.
Next, we examine the performances on the 1000D L ASSO -H ARD and the 300D L ASSO -H
IGH thatexhibit active subspaces, here without observational noise.
BA X US achieves considerably bettersolutions than all state-of-the-art methods.
We also note that T U RBO and CMA-ES perform betterthan SAASBO and A LEBO .
While one may expect BA X US to outperform T U RBO and CMA-ESon these tasks with high input dimensions, it is surprising that SAASBO, H E SBO, and A LEBO arenot able to benefit from the present active subspace.
Here BA X US’s strategy to adaptive expand thenested subspace is superior.
Another crucial observation is that performances of BA X US vary onlyslightly across runs.
Thus, BA X US is robust despite the stochastic construction of the embedding.
Across the broad collection of benchmarks, BA X US is the only method to achieve constantly highperformance.
Noisy benchmarks.
We evaluate the algorithms also for tasks with observational noise.
Fig.
4summarizes the results.
We observe that BA X US achieves considerably better solutions for anynumber of observations than the competitors.
Moreover, we note that the performances of SAASBO,
CMA-ES, and H E SBO ( d = 20 ) degrade considerably on the L ASSO -H IGH task compared to the noise-free formulation of the task studied above.
BA X US’s performance is equally strong as for the noise-free case and keeps making progress after 1000 observations.
BA X US embedding ablation study.
To investigate whether the proposed family of nested randomsubspaces contributes to the superior performance of BA X US, we replaced the new embedding with asimilar family of nested H E SBO embeddings.
The results show that the proposed embedding providesa significant performance gain.
Due to space constraints, the results were moved to Appendix A.
Discussion
High-dimensional Bayesian optimization is aspiring to unlock impactful applications broadly inscience and industry.
However, state-of-the-art methods suffer from limited scalability or, in somecases, require practitioners to ‘guess’ certain hyperparameters that critically impact the performance.
This paper proposes BA X US that works out-of-the-box and achieves considerably better performancefor high-dimensional problems, as the comprehensive evaluation shows.
A key idea is to scale upthe dimensionality of the target subspace that the algorithm optimizes over.
We apply a simplestrategy that we find to work well across the board.
However, we expect substantial headroom intailoring this strategy to specific applications, either using domain expertise or by a more sophisticateddata-driven approach that, for example, learns a suitable target space.
Moreover, future work willexplore extending BA X US to structured domains, in particular the combinatorial spaces common inmaterials sciences and drug discovery.
Societal impact.
Bayesian optimization has recently become a popular tool for tasks in drugdiscovery [48], chemical engineering [10, 26, 29, 54, 56], materials science [22, 27, 28, 49, 59, 63],aerospace engineering [2, 37, 41], robotics [11, 12, 39, 43, 51], and many more.
This speaks to theprogress that Bayesian optimization has made in becoming a robust and reliable ‘off-the-shelf solver.’
However, this promise is not yet fulfilled for the newer high-dimensional Bayesian optimization fieldthat allows optimization over hundreds of ‘tunable levers.’
The applications mentioned above benefitfrom incorporating more such levers in the optimization: it allows for more detailed modeling ofan aerospace design or a more granular control of a chemical reaction, to give some examples.
Theevaluation shows that the performance of state-of-the-art methods degenerates drastically for suchhigh dimensions if the application does not meet specific requirements.
Adding insult to the injury,such requirements like the dimensionality of an active subspace cannot be determined beforehand.
The proposed algorithm achieves a robust performance over a broad collection of tasks and thus willbecome a ‘goto’ optimizer for practitioners in other fields.
Therefore, we release the BA X US code.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes] See Section 5.
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] SeeSection 5. (d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes]
(b) Did you include complete proofs of all theoretical results?
[Yes] See Appendix C.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes]
(b) Did you specify all the training details (e g., data splits, hyperparameters,
how they
were chosen)?
[Yes]
(c) Did you report error bars (e g., with respect to the random seed after running experi-
ments multiple times)?
[Yes]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] See Appendix F.
If you are using existing assets (e g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] See Appendix F.
(b) Did you mention the license of the assets?
[Yes] See Appendix F.
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
[Table] (e) Did you discuss whether the data you are using/curating contains personally identifiableinformation or offensive content?
[N/A]
[Table]
If you used crowdsourcing or conducted research with human subjects...
[Table] (a) Did you include the full text of instructions given to participants and screenshots, if [Table]
applicable?
[N/A]
[Table] (b) Did you describe any potential participant risks, with links to Institutional Review [Table]
Board (IRB) approvals, if applicable?
[N/A]
[Table] (c) Did you include the estimated hourly wage paid to participants and the total amount [Table]
spent on participant compensation?
[N/A]
[Table]
A Ablation study for the BA X US embedding
We conduct an ablation study to investigate the difference between the BA X US and H E SBOembeddings.
We run T U RBO of Eriksson et al. [21] in an embedded subspace with the two differentembeddings.
We use a version of A CKLEY 10 (ten effective dimensions, i.e., d e = 10 ), where weshift the optimum away from the origin with a uniformly random vector δ ∈ [ − 32 .
768] d ewith δ i ∼ U ( − 32 .
The function we optimize is thenf ShiftedAckley
( x ) =
f Ackley10 ( x + δ ) .
We adjust the boundaries of the search space such that f ShiftedAckley10 is evaluated on the domain
X = [ − 32 .
768] d e .
The reason for shifting the optimum is that the original A CKLEY functionhas its optimum at the origin.
In that case, any count-sketch type embedding contains this optimum,even if all the important input dimensions are mapped to the same target dimension.
We add 10 dummy dimensions, such that D = 30 and set d = 20 .
With this problem-setting, the
BA X US and H E SBO embeddings have a probability of approximately 0.27 and 0.07 to contain theoptimum, respectively.
The BA X US embedding has a higher probability to optimum distribution
The left side of Figure 5 shows the incumbent mean for T U RBO in the two different embeddedsubspaces.
The shaded regions show one standard error.
T U RBO in an BA X US embedding hassignificantly better optimization performance than in a H E SBO embedding.
The right side of Figure 5shows the distributions of the final incumbents and their median.
The BA X US embedding leads to asignificantly lower median and only rarely a similarly bad embedding as the H E SBO method, whencombined with T U RBO.
We perform a two-sided Wilcoxon rank-sum statistical test to check the difference between the bestobserved function values for the two embeddings.
The difference is significant with p ≈ 0 .
The performance difference between the two embeddings depends on the characteristics of thefunction and the different dimensionalities, the input dimensionality D , the target dimensionality d ,and the effective dimensionality d e .
For problems with a few effective dimensions and many inputdimensions, the BA X US and H E SBO embeddings become more similar (see Figure 2).
However, by
Corollary 2, the BA X US embedding is, in expectation and in terms of success probability, alwaysbetter than the H E SBO embedding for arbitrary functions with an axis-aligned active subspace.
For functions with an optimum at the origin, both embeddings contain that optimum regardless of d : Even if all important input dimensions are mapped to the same target dimension, the optimum in theinput space can be reached by “setting” this particular target dimension to zero.
The distribution of important input dimensions.
The success probability studied in Section 3and above only considers the case where all the important input dimensions are mapped to differenttarget dimensions.
It might not be necessary to map all the important input dimensions to differenttarget dimensions, but it is generally beneficial if they spread out.
For example on M OPTA 08 or
S VM in Figure 3 (top left, top center), BA X US makes initial progress, then seems to be in a too lowembedding before the target dimensionality is increased and BA X US makes further progress.
If onlythe case where all d e important input dimensions are mapped to individual target dimensions allowedto make progress, BA X US would not make any progress until it reaches a suitable embedding.
Even though it is often possible to make progress without mapping all important input dimensions toindividual target dimensions, it is undoubtedly beneficial if, for a given target dimensionality, theymap to more individual target dimensions; The more the important input dimension spread out, themore the effective subspace can be covered.
Figure 6 shows a histogram of the number of individual target dimensions the d e = 10 importantinput dimensions map to in the example from above for the BA X US and H E SBO embeddings.
The bars on the very right approximately give the success probability for the H E SBO and BA X USembeddings - they correspond to a success (see Definition 2).
The BA X US embedding maps the 10important input dimensions to 9 or 10 individual target dimensions in around 74% of the cases; in the
H E SBO embedding, this happens in less than half as many cases ( 22% ).
T U RBO with BA X US embedding vs. BA X US.
We compare the simple idea of running T U RBOin a BA X US embedding with the BA X US algorithm as described in Section 3.
We run this simpleapproach for 11 different target dimensionalities d ( 2 , 10 , 20 , 30 , 40 , 50 , 60 , 70 , 80 , 90 , 100 ) on the
L ASSO -H ARD benchmark and show the results with a sequential color map in Figure 7.
Only thefirst d = 2 -dimensional embedding achieves the same initial speedup as BA X US, which is expectedas BA X US starts in a similarly low-dimensional initial embedding.
However, the fixed embeddingcannot explore the input space sufficiently and has the worst final solution.
High-dimensional fixedembeddings have more freedom in exploring the input space; however, they suffer from slower initialoptimization performance.
BA X US has the same initial speedup as the two-dimensional fixed embedding but can explore thespace further by increasing the dimensionality of its embedding.
Summing up, we observe that BA X US achieves a better performance than T U RBO with a fixedembedding dimensionality.
B Additional Lasso benchmark
In addition to the synthetic L ASSO -H IGH and L ASSO -H
ARD benchmarks studied in Section 4, weevaluate BA X US on the L ASSO -DNA benchmark from L ASSO B ENCH [55].
The L ASSO -DNAbenchmark is a biomedical classification task, taking binarized DNA sequences as input [55].
Figure 8 shows the mean performance of BA X US on the L ASSO -DNA.
Each line shows the incumbent mean; the shaded regions around the lines show one standard error.
We see the same qualitative behavior as discussed in Section 4: BA X US reaches a good initial solution faster than any other method and converges quickly.
After a worse start, T U RBO finds slightly better solutions than BA X US.
C Theoretical foundation for the BA X US embedding
For reasons of clarity and comprehensibility we first recapitulate Definition 1 from Section 3.
Definition 1 ( Count-sketch type embedding ) .
We say that a matrix S ∈ { 0 , ± 1 } D × d is of thecount-sketch type if and only if each column in S has exactly one non-zero entry.
We state the following two definitions.
Definition 3 ( Optima-preserving count-sketch type embedding ) .
An embedding of the countsketch type is optima-preserving if each target dimension contains at most one effective inputdimension.
Definition 4 ( Function with an (axis-aligned) active subspace ) .
Let X = [ − 1 , 1] D . A functionf : X → R has an active subspace , if there exist a subspace (i.e., a space Z ⊆
R d e , with d e < Dwhere d e ∈ N ++ is the effective dimensionality) and a projection matrix S ⊺ ∈ R d e × D , such thatfor any x ∈ X there exists a z ∈ Z so that f ( x ) =
f ( S ⊺ z ) and d e is the smallest integer withthis property.
The subspace Z is further called axis-aligned if S ⊺ is of count-sketch type and
Z = [ − 1 , 1] d e .
C.1 Proof of Lemma 1
We prove the success probability for the BA X US embedding.
For this, we first re-state Definition 2from Section 3.
Definition 2 ( Success of a count-sketch type embedding ) .
A success of count-sketch type embedding is the event Y ∗ =
“For any function f : X → R with an axis-aligned active subspace thatattains minima in X , there exists a y ∗ in Y such that S ⊺
y ∗ ∈ arg min x ∈X f ( x ) ”, where S ⊺ mapsfrom the target space Y to the input space X .
Lemma 1 ( Success probability of the BA X US embedding ) .
Let D be the input dimensionality andd ≥ d e the dimensionality of the embedding.
Let β small = (cid:4) Dd (cid:5) and β large =
(cid:6) Dd (cid:7) be the small andlarge bin sizes.
Then, the probability of Y ∗ (see Definition 2) with the BA X US embedding is
[Equation]
Proof.
The assignment of input dimensions to target dimensions and the signs of the input dimensionsfully define the H E SBO and BA X US embeddings.
Note that the signs do not have any effect onp A ( Y ∗ ; D, d, d e ) because they only correspond to “flipping” the input dimension in the target spaceand our construction ensures that the value ranges are symmetric to the origin.
An assignment is optima-preserving, if and only if it is possible to find a point in Y that maps to anoptimum in X for any f .
The “only if” is true because f is assumed to have an axis-aligned activesubspace with d e effective dimensions.
This means that the optima in X only change their functionvalues along the d e effective dimensions.
Suppose it is possible to find a point in Y that maps to anarbitrary optimum in X .
In that case, the assignment has to be optima-preserving because it has tobe able to adjust all the d e effective dimensions in X individually.
However, this generally requireseach effective input dimension to be mapped to a different target dimension (note that we requirebeing able to represent the optimum for any f ).
Otherwise, there would be at least two effective inputdimensions that cannot be changed independently.
Therefore, the probability of Y ∗ is equal to theprobability of an optima-preserving assignment.
As all assignments are equally likely under the construction, the probability of an assignment to beoptima-preserving is equal to the number of possible optima-preserving assignments divided by thetotal number of assignments.
There are (cid:0) Dd e (cid:1) ways of distributing the d e effective dimensions acrossthe D positions (denominator in Eq.
Let us first assume that β small
= β large := β , i.e., all target dimensions have the same number ofinput dimensions and d divides D .
There are (cid:0) dd e (cid:1) ways of distributing the d e effective dimensionsacross the d different target dimensions.
Given one effective dimension, there are β ways in whichthis dimension can map to the target dimension.
Therefore, for the case of equal bin sizes, the successprobability is given by
[Equation]
[Figure]
Next, we generalize Eq.
(6) for cases where d does not divide D .
In that case, there are two bin sizes:β small and β large with β large = β small + 1 . There are dβ large − D small bins (i.e., bins with bin sizeβ small ) and D − dβ small large bins: D − dβ small gives the number of input dimensions that wouldnot be covered if all bins were small.
Since β small and β large differ by 1 , this also gives the number ofbins that have to be large.
Conversely, if we would only have large bins, we would cover dβ large − Dtoo many input dimensions.
Therefore, we need two large and one small bin (See Figure 9). Next, we need to consider all ways of distributing the d e effective dimensions across the the dβ large − D small and D − dβ small large bins (numerator in Eq. 5). Note that when we assign more effectivedimensions to small or large bins than there are such bins, the binomial coefficient becomes zeroand that case does not contribute to the sum.
For each bin containing an effective dimensionality,there are β small (or β large ) ways this dimension can lie in the bin (compare Figure 1).
This gives theβ -factors in the numerator.
It is easy to see that Eq.
(5) is equivalent to the uniform formulation in Eq.
(6) when d divides
D .
Consider the fact that (cid:0) 00 (cid:1) =
1 and (cid:0)
x (cid:1) = 0 for all x ∈ N ++ .
Then, when d divides D ,β small = β large = β , d (1 + β small )
− D = d , and D − dβ small = 0 .
Therefore, the successprobability for the almost-uniform case is given by
[Equation]
where the last equality is true because the sum is zero unless i = d e .
C.2 Proof of Corollary
We prove the equivalence of the BA X US and the H E SBO embeddings in terms for success probability.
Corollary 1.
For D → ∞ , the success probability of the BA X US embedding islim
[Equation]
Proof.
The H E SBO embedding is constructed by choosing a target dimension and a sign uniformlyat random for each input dimension.
With this construction, the H E SBO embedding is independentof D .
Letham et al.
[38] give an expression for the H E SBO success probability:
[Equation]
The validity of this expression can be seen by considering the probability that d e effective inputdimensions are mapped to d target dimensions without collision if d ≥ d e :
[Equation]
Clearly, as D → ∞ , the H E SBO embedding has equal bin sizes; intuitively, the H E SBO embeddingand our formulation are equivalent in the limit.
We consider lim β →∞ , which is equivalent to lim D →∞ as β =
Dd and d is fixed.
We further use theuniform formulation in Eq.
(6) as a starting point because all bins are of infinite size in the limit( (cid:4)
Dd (cid:5) =
(cid:6) Dd (cid:7) for D → ∞ , d ∈ N ++ ).
[Equation]
Eq.
(17) is the success probability of the H E SBO embedding [38].
C.3 Proof of Corollary
We prove the optimality of the BA X US embedding in terms of success probability.
Corollary 2.
With the same input, target, and effective dimensionalities ( D , d , and d e ), no countsketch type embedding has a higher success probability than the BA X US embedding.
Proof.
By Definition 1, an embedding matrix S ∈ { 0 , ± 1 } D ×
d is of the count-sketch type if eachrow in S has exactly one non-zero entry.
Such an embedding can always be interpreted as disjointsets of signed input dimensions that are assigned to different target dimensions:
For the n -th inputdimension, find the column where the non-zero entry lies.
The number of the respective column gives the target dimension; the entry in the matrix itself gives the sign.
Conversely, each target dimensionhas a set of contributing input dimensions, and we call the set of input dimensions mapping to a targetdimension a “bin”.
The sign of the input dimensions does not influence the success probability as itdoes not influence the ability of an embedding to contain the optimum.
We will prove that the BA X US embedding is optimal because every other embedding of the countsketch type has a success probability that is lower or equal to the success probability of the BA X USembedding.
We start the proof by giving the success probability for arbitrary bin sizes.
Let [-i 
-]β n be the bin size of the n -th bin. By Definition 2, a success occurs if each bin contains at mostone effective input dimension. The success probability for arbitrary bin sizes therefore has to considerthe number of cases where each bin contains at most one effective input dimension and the number ofbins containing one effective input dimensions is equal to the number of effective input dimensionsd e .
In a bin of size β i , the effective input dimension can lie in β i different locations. Therefore,for each bin containing an effective input dimension, we have to consider all locations in which theimportant input dimension can lie. Bins not containing an effective dimension do not contribute tothe success probability.
By this reasoning, the success probability for arbitrary bin sizes is given by
[Equation]
with β n > 0 , (cid:80) dn =1 β n = D , and d ≥ d e : First, we require β n > 0 w.l.o.g.
as bins not containing any input dimensions cannot influencethe success probability. For each n from 1 to d , the value i n indicates whether the n -th bin (ortarget dimension) contains an effective dimension ( i n = 1 ) or not ( i n = 0 ). The indicator variable 745( (cid:80) dn =1 i n )=
d e ensures that only cases where exactly d e bins contain an effective input dimension areconsidered. Note that (cid:80) 1 i 1 =0 (cid:80) 1 i 2 =0 . . . (cid:80) 1 i d =0 1 ( (cid:80) dn =1 i n )=
d e = (cid:0) dd e (cid:1) . For each case where the d eeffective dimensions are assigned to d e out d bins, the term (cid:81) dn =1 β i n n accounts for the locations inwhich the effective dimension can lie in the n -th bin. The exponent ensures that only bins actuallycontaining an important dimension contribute to the denominator.
As in Lemma 1, the numeratorof Eq.
(18) gives all ways of assigning d e effective dimensions to D input dimensions. Note thatfor the case where β i = β j for all i, j ∈ { 1 , . . . , d } , Eq. (18) is equal to Eq.
For the case whereβ i ∈ { β s , β l } as defined in Section , Eq.
(18) is equal to Eq.
We now prove that any count sketch embedding has a success probability that is less or equal tothe success probability of the BA X US embedding. Let β small , β large , and p A ( Y ∗ ; D, d, d e ) as in Lemma 1. Then,
[Equation]
By Maclaurin’s inequality, the following holds for the term (cid:81) dn =1 β i n n in the numerator of Eq. 18:(cid:81)
dn =1 β i n n ≤ β d e .
The numerator in Eq.
(18) can also be written as E d e ( β 1 , . . . , β d ) (cid:0) dd e (cid:1) where
[Equation]
de ) is the d e -th elementary symmetric mean and e d e ( β 1 , . . . , β d ) is thed e -th elementary symmetric polynomial of β 1 , . . . , β d . Then, by Maclaurin’s inequality
[Equation]
20) with equality if and only if β i = β j for i, j ∈ { 1 , . . . , n } .
However, d does not divide D , β is no integer which is impossible.
Observe that (cid:81) dn =1 β i n n is aconvex function because all factors in the product are in N ++ .
Therefore, the more the bin sizesdiffer from the “mean” β = Dd , the lower the success probability.
If d does not divide D , β large andβ small have the smallest possible difference 1. Otherwise, β i small β d e − i large = β d e .
Therefore, the BA X US embedding is optimal in terms of success probability.
D Consistency of BA X US
We prove the global convergence of function values for BA X US.
The proof idea is similar to Erikssonand Poloczek [20] but relaxes the assumption of a unique global minimizer.
By construction, f hasan axis-aligned active subspace (see Definition 4), i.e., there exists a set of dimensions of f that donot influence the function value. Thus, an optimal solution stays optimal regardless how non-activedimensions are set.
This is why we have to relax the assumption of a unique global minimizer in theinput space. Instead we assume a unique global minimizer in the active subspace z ∗ ∈ Z that canmap to arbitrarily many minimizers in the input space.
Theorem 2 ( BA X US convergence ) .
With the following definitions:
D1. { x } ∞ k =1 is a decreasing sequence of incumbent function values;D2. x ∗ ∈ arg min x ∈X f ( x ) is a minimizer in X ; and under the following assumptions:
A1.
D is finite; A2. f is noiseless;
A3. The objective f has an axis-aligned active subspace and is bounded in X , i.e., ∃ C ∈ R ++s.t.
| f
( x ) | < C ∀ x ∈ X ; A4.
At least one of the minimizers x ∗ i lies in a continuous region with positive measure; A5. Once BA X US reached the input dimensionality D , the initial points { x i } n init i
=1 after each TR
restart for BA X US are chosen such that ∀ δ ∈ R ++ and x ∈ X ∃ ν ( x , δ ) >
P ( ∃ i : || x − x i || 2 ≤ δ ) ≥ ν ( x , δ ) , i.e., the probability that at least one point in { x i } n init i =
ends up in a ball centered at x with radius δ is at least ν ( x , δ ) ;
f ( x k ) converges to f ( x ∗ ) with probability
Proof.
We first show that BA X US must eventually arrive at an embedding that is equivalent to theinput space. By Assumption A1, the fail tolerance (i.e., the number of times BA X US needs to fail infinding a better solution until the TR base length is shrunk) is always finite since it is always boundedby the target dimension ( ∀ i τ i fail ≤ d i ) which is at most equal to D ( ∀ i d i ≤ D ). By the facts that
BA X US considers any sampled point an improvement only if it improves the current best solution byat least some constant γ ∈ R ++
and that f is bounded (Assumption A3), BA X US can only performa finite number of function evaluations without increasing the target dimensionality of its embedding.
Once BA X US reaches D , it behaves like T U RBO [21] for which Eriksson and Poloczek [20] provedglobal convergence assuming a unique global minimizer. For the case d e < D , we notice thatmultiple minima in the input space occur due to non-active dimensions that do not influence thefunction value.
The remainder of our proof is based on the convergence theorem for global search by Solis and Wets[58] which proves convergence of function values for random search with possibly multiple minima.
We note that considering the sequence of incumbent points (Definition D1) and the fact that, at each
TR restart, BA X US performs random restarts with uniform probability on X , then BA X US satisfiesthe assumptions of the theorem. The Solis and Wets [58] theorem states that for a sequence { x k } ∞ k =1 of sampling points with ε ∈ R ++ [Equation] where R ε = { x ∈ X : f ( x ) <
α + ε } is the set of ε -optimal function values, α = inf { t : v ( x ∈ X : f ( x ) < t ) > 0 } is the essential infimum , and v is the Lebesgue measure. Note that the essentialinfimum α is equal to the minimum if the minimizer lies in a continuous region of positive measure,i.e., α = f ( x ∗ i ) .
By Assumption A4 and by letting ε → 0 , f ( x k ) converges to f ( x ∗ i ) .
E The nested family of random embeddings
We describe the method for increasing the target dimensionality under the retention of the observations.
Suppose that we have collected n observations and are in target dimension d when Algorithm 2 isinvoked.
Algorithm 2 loops over the target dimensions 1 , . . .
, d .
For each target dimension, thecontributing input dimensions are randomly re-assigned to new bins of given sizes.
This can, forexample, be realized by, first, randomly permuting the list of contributing input dimensions, and,secondly, dividing the list into b + 1 chunks (bins).
If the number of contributing input dimensions isless than b + 1 (remember that b is the number of new bins), then it is not possible to re-assign thecontributing input dimensions to b + 1 bins.
Therefore we re-assign the contributing input dimensionsto ˆ b = min( b, l s − 1) bins, where l s is the number of contributing input dimensions to the s -th targetdimension.
This also ensures that the target dimension never grows larger than D in the BA X USembedding.
We evenly distribute the l s contributing input dimensions across the ˆ b bins by againusing the BA X US embedding.
This gives a smaller projection matrix ˜ S ⊺ which we finally use toupdate S ⊺ :
Input: transposed embedding matrix S ⊺ , number of new bins per latent dimension b , observed points
Y ∈ [ − 1 , 1] n × σ dimensions ∈
D
BAxUS for do if ˜ S ⊺ ij ̸
then if j > 1 then
S
[-˜ S ⊺ ij ̸ = 
 then if j > 1 then Move values that new bins to end of S ⊺ . S ⊺ D si , ˆ d − ˆ b − 1+ j ← σ i columns-][Equation]
S
[Equation] ←
Return S ⊺ and Y .
F Additional implementation details
We benchmark against SAASBO, T U RBO, H E SBO, A LEBO , and CMA-ES:•
For
SAASBO, we use the implementation from [19] ( https://github.com/
martinjankowiak/saasbo , license: none, last accessed: 05/09/2022)
• For T U RBO, we use the implementation from [21] ( https://github.com/uber-research/TuRBO , license: Uber , last accessed: 05/09/2022).
• For H E SBO and A LEBO , we use the implementation from [38] ( https://github.com/facebookresearch/alebo , license: CC BY-
• For the L ASSO benchmarks, we use the implementation from [55] ( https://github.com/ksehic/LassoBench , license: MIT and BSD-3-Clause , last accessed: 05/09/2022).We use GPyTorch to train the GP with the following setup: We place a top-hat prior one the Gaussianlikelihood noise, the signal variance, and the length scales of the Matérn 5/2 ARD kernel.
The intervalfor the noise is [0 . 005 , 0 . 2] , for the signal variance [0 . 05 , 20] , and for the lengthscales [0 .
005 , 10] .We evaluate on the synthetic B RANIN 2 2 and H ARTMANN 6 3 functions.
Since we augment the functionwith dummy dimensions, we use the same domain for x 1 and x 2 , namely [ − 5 , 15] D for B RANIN 2and [0 , 1] D for H ARTMANN 6.Similar to T U RBO, we sample a min(100 d n , 5000) -element
Sobol sequence on which we minimizethe posterior sample.
To maximize the marginal log-likelihood of the GP, we sample 100 initialhyperparameter configurations.
The ten best samples are further optimized using the A DAM optimizerfor 50 steps.
We ran the experiments for approximately 7 , 000 core hours on Intel Xeon Gold 6130 CPUs providedby a compute cluster