Increasing the Scope as You Learn: Adaptive Bayesian Optimization in Nested Subspaces
Anonymous Author(s)
AffiliationAddress email
Abstract
Recent advances have extended the scope of Bayesian optimization (BO) toexpensive-to-evaluate black-box functions with dozens of dimensions, aspiring tounlock impactful applications, for example, in the life sciences, neural architecturesearch, and robotics.
However, a closer examination reveals that the state-of-the-artmethods for high-dimensional Bayesian optimization (HDBO) suffer from degrading performance as the number of dimensions increases or even risk failure ifcertain unverifiable assumptions are not met.
This paper proposes BA X US thatleverages a novel family of nested random subspaces to adapt the space it optimizesover to the problem.
This ensures high performance while removing the risk offailure, which we assert via theoretical guarantees.
A comprehensive evaluationdemonstrates that BA X US achieves better results than the state-of-the-art methodsfor a broad set of applications.
Introduction
The optimization of expensive-to-evaluate black-box functions where no derivative information isavailable has found many applications, for example, in chemical engineering [8, 24, 27, 52, 54],materials science [20, 25, 26, 28, 47, 57, 61], aerospace engineering [35, 39], hyperparameteroptimization [3, 29, 34, 55], neural architecture search [32, 51], vehicle design [11, 30], hardwaredesign [16, 44], drug discovery [46], robotics [9, 10, 37, 41, 49], and the life sciences [15, 53, 58].
Increasing the number of dimensions (or parameters) of the optimization problem usually allows forbetter solutions.
For example, for a chemical reaction, we obtain a more granular control by exposingmore process parameters; for a design problem, we may optimize a larger number of design decisionsjointly; in robotics, we gain access to more sophisticated control policies.
A series of breakthroughsfacilitated the increasing adoption of Bayesian optimization in other domains that have pushed theenvelope of high-dimensional Bayesian optimization.
Methods that rely on local ’trust regions’usually achieve good performance if the input dimensionality of the problem does not exceed a coupleof dozens.
However, we observe that their performance degrades for higher-dimensional problems.
This is not surprising, given that the surrogate model of the trust regions has the full dimensionalityof the problem.
Other state-of-the-art methods suppose the existence of a low-dimensional activesubspace and enjoy great scalability if they find such a space.
The caveat is that its existence isusually not known for practical applications.
Moreover, the user needs to ’guess’ a good upper boundon its dimensionality to enjoy a good sample efficiency.
In this work, we propose a theoretically-founded approach for high-dimensional Bayesian optimization, BA X US ( B ayesian optimization with A daptive a x is-aligned U niform S ubspaces), that reliablyachieves a high performance on a comprehensive set of applications.
BA X US utilizes a family ofnested embedded spaces which allows increasing the dimensionality of the domain that it currentlyoptimizes over as it collects more data to refine its surrogate.
As a byproduct, BA X US can leverage an active subspace, if it exists, without requiring the user to ’guess’ its dimensionality.
BA X US isbased on a novel random linear subspace embedding that enables a more efficient optimization andhas strong theoretical guarantees.
We make the following contributions:
We develop BA X US, a reliable algorithm for high-dimensional BO that scales to higher inputdimensions than the state-of-the-art.2.
A novel family of nested random embeddings that has the following properties: a)
The BA X USembedding has a strictly higher probability of containing an optimum than the H E SBO embeddingproposed by Nayebi et al.
b)
The BA X US embedding the optimal embedding of the countsketch type, as defined in Def.
c) its probability containing an optimum converges to the one ofthe H E SBO embedding as the input dimensionality D → ∞ .3.
A comprehensive evaluation on a representative collection of benchmarks that demonstrates thatBA X US outperforms the state-of-the-art methods.
The remainder of this paper is structured as follows.
Section 2 states the problem and discussesrelated work.
Section 3 presents the BA X US algorithm and the corresponding embedding.
Section 4evaluates BA X US on a variety of benchmarks.
Lastly, we discuss our work in Section 5.
Background
The task is to find a minimizer
[Equation]
where X = [ − 1 , +1] D .
The objective function f : X → R is an expensive-to-evaluate black-boxfunction; hence the number of function evaluations needed to find an optimizer is crucial.
Evaluationsmay be subject to observational noise, i.e., f ( x i ) + ε i , with ε i ∼ N (0 , σ 2 ) .
The focus of this workis on scalable high-dimensional Bayesian optimization , where the input dimensionality D is in thehundreds, and the sampling budget may comprise a thousand or more function evaluations.
Linear embeddings.
A successful approach for high-dimensional Bayesian optimization is toassume the existence of an active subspace [14], i.e., there exist a space Z ∈ R d e , with d e < D anda function g : Z → R , such that for all x : g ( z ) ≈ f
( x ) where z is a projection of x in Z and d e isthe effective dimensionality of the problem.
In practice, both d e and Z are unknown.
REMBO (Random embedding BO) [63] and H E SBO (Hashing-enhanced subspace BO) [45] tryto capture this active subspace by a randomly chosen subspace.
Therefore, they generate a randomprojection matrix S ⊺ that maps from a d -dimensional subspace Y ∈ R d with d ≪ D (the targetspace ) to X .
We call d the target dimensionality .
For REMBO, each entry in S ⊺ is normallydistributed.
REMBO uses a heuristic to determine a hyperrectangle in Y that it then optimizesover.
Note that the bounded domain may not contain a point that maps to an optimizer of f , arisk aggravated by distortions introduced by the projection.
[4, 5, 6] proposed ideas to mitigatethe issue.
H E SBO’s random projection assigns one target dimension and sign ( ± 1 ) to each inputdimension.
This embedding is inspired by the count-sketch algorithm [12] for estimating frequentitems in data streams.
The sparse projection matrix S ⊺ is, except for the signs, binary, and each rowhas exactly one non-zero entry.
Even though this embedding avoids REMBO’s distortions, it hasa lower probability of containing the optimum [36].
A LEBO [36] uses a Mahalanobis kernel andimposes linear constraints on the acquisition function to avoid projecting outside of X .
Non-linear embeddings.
Several works use autoencoders to learn non-linear spaces for optimization, trading in sample efficiency.
Tripp et al.
[60] change the training objective of a variationalautoencoder (VAE) [33] to make the target space more suitable for optimization.
They give higherweight to better-performing points when training the variational autoencoder (VAE) and show thatthis improves optimization.
Moriconi et al.
[42] incorporate the training of an autoencoder directlyinto the likelihood maximization of a Gaussian process (GP) surrogate.
The computational cost iscubic in the number of samples and the input dimension.
Lu et al.
[38] and Maus et al.
[40] useautoencoders to learn embeddings of highly structured input spaces such as kernels or molecules.
Other approaches include partial least squares [7] or sliced inverse regression [13].
High-dimensional BO in the input space.
One method to make HDBO in the input space feasibleis trust regions (TRs) [19, 48, 50, 66].
The T U RBO algorithm [19] optimizes over bounded TRsinstead of the global space, adapting their side lengths and the center points during the optimizationprocess.
By restricting function evaluations to trust regions, T U RBO addressed the problem of overexploration; see [19] for details.
Note that the TRs have full input dimensionality, which may impact
T U RBO’s ability to scale to very large dimensions.
Nonetheless, T U RBO set a new state-of-the-artby scaling to dozens on input dimensions and thousands of function evaluations.
Wan et al.
[62]extended the idea of TRs to categorical and mixed spaces by using the Hamming distance to define the
TR boundaries.
SAASBO [17] uses sparse priors on the GP length scales which seems particularlyvaluable if the active subspace is axis-aligned.
Indeed, SAASBO can outperform T U RBO on certainbenchmarks [17].
The cost of inference scales cubically with the number of function evaluations;thus, SAASBO is not expected to scale beyond small sampling budgets, which is confirmed by ourexperiments.
Another line of research relies on the assumption that the input space has an additivestructure [21, 31, 43, 64].
Additive GPs rely on computationally expensive sampling methods tolearn a decomposition of the input variables, which limits the scalability of such methods to problemsof moderate dimensionalities and sampling budgets [19, 45].
In Sect.
4 we evaluate the performances of T U RBO, SAASBO, A LEBO , and H E SBO.
Moreover,we study the popular CMA-ES [22] and random search [2].
The BA X US algorithm
Wang et al.
[63] showed that the REMBO embedding can represent the optimum in the target spaceif d ≥ d e and there are no bounds on the target and input spaces, i.e., Y = R d and X = R D .
If thereis a point in the target space that maps to an optimum in the input space for arbitrary functions fattaining an optimum in X , then Y contains that optimum.
For d < d e , it is generally impossible torepresent an optimum in X for arbitrary f as S ⊺ projects to a d -dimensional subspace in X .
We callthe probability of a target space to contain the optimum the success probability .
For d ≥ d e , there isa positive success probability that increases with d [36, 63].
The main problem is to set d as small aspossible to avoid the detrimental effects of the curse of dimensionality but as large as necessary toachieve a high probability of Y to contain an optimum.
In reality, the effective subspace and its dimensionality are usually unknown.
The performance ofmethods, such as REMBO [63], H E SBO [45], and A LEBO [36] depends on choosing d such that thesuccess probability is high.
Therefore, they implicitly rely on guessing the effective dimensionalityd e appropriately.
We argue that choosing the target dimensionality is problematic in many practicalapplications.
If chosen too small, the subspace cannot represent f sufficiently well.
If it is chosen toolarge, the curse of dimensionality slows down the optimization.
The proposed algorithm, BA X US, operates on target spaces of increasing dimensionality whilepreserving previous observations.
Let d init be the initial target dimensionality and m the totalevaluation budget.
BA X US starts with a d init -dimensional
embedding that is increased over timeuntil, after m D ≤ m evaluations, it roughly reaches the input dimensionality D .
With this strategy,we can leverage the efficiency of BO in low-dimensional spaces while being guaranteed to find anoptimum in the limit.
Increasing the target dimensionality is enabled by our embedding, which letsus convey observations from previous, lower-dimensional target spaces into more high-dimensionaltarget spaces.
We further use a TR-based approach based on Eriksson et al.
[19] to carry outoptimization for high target dimensions effectively.
BA X US uses a GP surrogate [65] to model thefunction in the target space.
Algorithm 1 gives the pseudocode for BA X US.
In Appendix D, weprove global convergence for BA X US.
We will now discuss the different components.
3.1 The Axis-aligned Uniform Subspace Embedding
The BA X US embedding is derived from the H E SBO embedding [45].
Given the input dimensionality
D and a target dimensionality d , the H E SBO embedding samples a target dimension ∈ { 1 , . . .
, d }and a sign ∈ {± 1 } for each input dimension uniformly at random.
Conversely, each target dimensionhas a set of signed contributing input dimensions.
We call the set of contributing input dimensions toa target dimension a bin .
These relations implicitly define the embedding matrix S ∈ { 0 , ± 1 } d × D ,where each column has exactly one non-zero entry [12].
In the H E SBO embedding, the number ofcontributing input dimensions varies between 0 and D .
S ⊺ = − +   +  − −   
Input space
Before Splitting
Target space - 0.
+ 0. + 0.
xxxxxyysplit 0.
Input space
After Splitting
Target space - 0.
+ 0. + 0.
xxxxxyyy0.
c opy
The interpretation of contributing input dimensions allows for an intuitive way to increase theembedding, which is shown in Figure 1.
We update the embedding matrix such that contributinginput dimensions of the target dimension are re-assigned to b + 1 bins.
We then say that we split thecorresponding target dimension.
Importantly, this type of embedding allows for retaining observations(see Figure 1) without any distortions.
Assume, for example, that y i is the dimension to be split.
The contributing input dimensions are re-assigned to y i and two new target dimensions y j and y k(here, b = 2 ); the observations can be retained by copying the observation for y i to y j and y k .
Theobservations remain undistorted in the input space because an input dimension re-assigned to a newtarget dimension still has the same value.
By doing so, the observations from the previous targetdimension get copied over to the new target dimension.
Algorithm 2 describes the procedure in detail.
In the BA X US embedding, we force each bin of a target dimension to have roughly the same numberof contributing input dimensions; the bin sizes differ by at most one.
First, we create a randompermutation of the input dimensions 1 , . . .
, D .
The list of input dimensions is split into min( d, D )individual bins.
If d does not divide D , not all bins can have the same size. We split the permutation D by
the input dimension is reached.
min
b Appendix E else discard of input dimensions such that the i -th bin has size ⌈ D/d ⌉ , if i + d ⌊ D/d ⌋ ≤ D , and ⌊ D/d ⌋ otherwise.
The first bins have one additional element with this construction if d does not divide D .
We furthersample a sign for each input dimension.
The sign of the input dimensions and their assignment totarget dimensions then implicitly define S ⊺ (see Figure 1).
We now show that target spaces of the BA X US embedding have a higher success probability than for the H E SBO embedding.
Definition 1 ( Count-sketch type embedding ) .
We say that a matrix S ∈ { 0 , ± 1 } D × d is of thecount-sketch type if and only if each column in S has precisely one non-zero entry.
Definition 2 ( Success of a count-sketch type embedding ) .
A success of count-sketch type embedding is the event Y ∗ =
“For any sparse axis-aligned function f : X → R that attains minima in X ,there exists a y ∗ in Y such that S ⊺ y ∗ ∈ arg min x ∈X f ( x ) ”, where S ⊺ maps from the target space Yto the input space X .
We refer to Definition 4 in Appendix C for a formal definition of a sparse axis-aligned function.
In Lemma 1 we give the success probability of the BA X US embedding.
All proofs have been deferredto Appendix C.
Lemma 1 ( Success probability of the BA X US embedding ) .
Let D be the input dimensionality andd ≥ d e the dimensionality of the embedding.
Let β small = ⌊ Dd ⌋ and β large =
Dd ⌉ be the small andlarge bin sizes.
Then, the probability of Y ∗ (see Definition 2) with the BA X US embedding is
[Equation]
Y [Equation] [Equation] Figure 2 shows three cases of the BA X US and H E SBO embedding’s success probabilities.
Thesuccess probability of the H E SBO embedding is given by p H ( Y ∗ ; d, d e ) =
d ! / (( d − d e )!
d d e ) [36].
Itis independent of D but is shown for varying d -ranges on the x -axis, therefore the visual discrepancy.
Except for when D → ∞ , the H E SBO embedding always has a non-zero probability of failing.
The
BA X US embedding ensures that the success probability is one whenever d ≥ D .
Discontinuities inthe curve of the BA X US embedding occur due to the unequal bin sizes in the BA X US embedding’ssuccess probability.
The difference between the two embeddings in Figure 2 is predominant when d eis high.
For small d e , both embeddings have a high success probability for moderate d .
If d e is high,for example d e = 20 , H E SBO requires d = 1000 to reach a success probability of approximately0 .
8 , while the BA X US embedding has a success probability of 1 as soon as d = D .
For finite D , thesuccess probability of the H E SBO embedding is lower.
Corollary 1.
In the limit, the success probability of the BA X US embedding is
[Equation]
and hence matches H E SBO ’s success probability
p H ( Y ∗ ; d, d e ) .
We now show that the BA X US embedding is optimal among count-sketch type embeddings.
Corollary 2.
With the same input, target, and effective dimensionalities ( D , d , and d e ), no countsketch type embedding has a higher success probability than the BA X US embedding.
3.2 Trust-region approach
Similar to T U RBO [19], BA X US operates in TRs.
TRs are hyper-rectangles in the input space; theirshape is defined by their base side length L and the GP length scales.
A TR always has volume L ,but the side length for each dimension is proportional to the corresponding length scale of the GPkernel.
The idea of this construction is that short GP length scales signify important dimensionsand reduce the extent of the hyper-rectangle in that dimension.
A TR is shrunk when T U RBO failsτ fail consecutive times to make progress.
If the algorithm consecutively makes τ success = 3 timesprogress, it expands the TR.
It restarts when the base side length L of the current TR falls below athreshold L min :
= 2 − 7 .
In that case, it discards all observations for the TR and initializes a new TRon new samples.
TRs enable T U RBO to focus on regions of the space that are close to the incumbent.
To choose the next evaluation point, T U RBO minimizes posterior samples of the GP on a set ofcandidates drawn from the TR (Thompson sampling [59]).
TR s are an essential component of BA X US as the target dimensionality grows exponentially.
We usethe same hyperparameter settings as T U RBO [19] with the following exceptions: First, we changethe criterion for when to restart a TR.
Instead of restarting a TR when it becomes too small, weincrease the target dimensionality by splitting each target dimension into several new bins unless BA X US had already reached the input dimensionality.
Secondly, we change the fail tolerance τ fail ,such that BA X US can roughly reach the input dimensionality in a fixed number of evaluations.
Themethod for this is described in Section 3.4.
We use the same algorithmic setup as T U RBO with theaforementioned exceptions.
3.3 Splitting strategy
Starting in a low-dimensional embedded space, BA X US successively increases the target dimensionality to increase the probability of containing the optimum.
By Corollary 2, it is optimal tokeep the distribution of contributing input dimensions in the target bins as uniform as possible.
Ateach splitting point, BA X US re-assigns the contributing input dimension of each target dimensionto b + 1 bins, where b is the number of new bins we create per target dimension, i.e., the targetdimensionality grows exponentially.
The number of splits required to reach some input dimensionality
D is logarithmic in D .
Compared to the constant growth of the target dimensionality, BA X US has alarger evaluation budget in each split and is less dependent on the value of D .
The algorithm starts intarget dimensionality d init .
After k splits, the target dimensionality is d k = d init ( b + 1) k .
3.4 Controlling the fail tolerance
As the dimensionality grows, the GP needs more data to model the objective function accurately.
Wepartition the evaluation budget into chunks of increasing sizes such that the sizes are proportional tothe target dimensionalities.
Each target dimensionality then has an individual evaluation budget.
Ourstrategy is to control the budget that BA X US has in each target dimensionality by the fail tolerance.
We choose this indirect approach as we do not want to force BA X US to increase the current targetdimensionality if it makes considerable progress.
The rule we propose is based on the fail toleranceτ fail required for the increasing embedding to reach the input dimensionality D , under the assumptionthat BA X US always fails in making progress.
We calculate the number of splits n required to reach
D by
[Equation]
with ⌊·⌉ indicating rounding to the nearest integer.
We assign each split i a split budget m s i that isproportional to d i , such that (cid:80) ni =1 m si = m D , where m D is the evaluation budget until D is reached:m si = (cid:106) − b · m D · d i 1 − ( b +1) n +1 (cid:109) .
With that budget, we set the raw adjusted fail tolerance in the i -th split as˜ τ i fail =
(cid:106) m si k (cid:107) , with k = (cid:106) log 1 / 2 L min L init (cid:107) being the number of TR shrinkages until the TR is restarted.
We aim to reduce the fail tolerance to enable BA X US to reach higher target dimensions whennecessary.
We are not interested in having even higher fail tolerances than the T U RBO algorithm, which could occur with this rule.
Therefore, we set the fail tolerance by taking theminimum of the raw adjusted fail tolerance and the T U RBO fail tolerances for single batches,τ i fail = max (cid:0) 1 , min (cid:0) ˜
τ i fail , d i (cid:1)(cid:1) , where d i is the target dimension in the i -th split.
Setting the initial target dimension.
If the target dimensionality is fixed, increasing the targetdimensionality at an exponential rate can lead to cases where none of the target dimensionalities d kare close to the input dimensionality D .
For example, for a 600-dimensional problem, with d init = 1 ,and b = 3 , the difference between d k and D is at least 344.
Since the final target dimensionalityof BA X US is at most D , such cases are in disagreement with the exponential growth of the targetdimensionality: Either the final target dimensionality is much less than the input dimensionality, orthe last split adds only a few new dimensions.
In the first case, we potentially reduce the successprobability.
In the latter case, we assign a too large budget to the final split.
Therefore, we set theinitial target dimensionality such that the final target dimensionality is as close to D as possible:
[Equation]
The optimal n is given by Eq.
Experimental evaluation
In this section, we evaluate the performance of BA X US on a 388D hyperparameter optimization task,124D design problem, and a collection of tasks that exhibit an active subspace.
The code is availableat hiddenforanonymization .
The experimental setup.
We benchmark against T U RBO [19], SAASBO [17], A LEBO [36],random search [2], CMA-ES [22], and H E SBO [45], using the implementations provided by therespective authors with their settings, unless stated otherwise.
For CMA-ES, we use the P Y C MA [23]implementation.
To show the effect of different choices of d , we run H E SBO with d = 10 andd = 20 . For A LEBO , we run the optimization with d = 10 .
We observed that A LEBO and SAASBOare constrained by their high runtime and memory consumption.
The available hardware allowed upto 100 evaluations for SAASBO and 500 evaluations for A LEBO .
Larger sampling budgets or highertarget dimensions for A LEBO resulted in out-of-memory errors.
We point out that limited scalabilitywas expected for these methods, whereas the other methods scaled to considerably larger budgets,as required for scalable BO.
We initialize each optimizer with ten initial samples and BA X US withb = 3 and m D = 1000 .
Plots show the mean performance with one standard error.
The benchmarks.
We evaluate the selected algorithms on six benchmarks that differ considerablyin their characteristics.
Following [63], we augment the B RANIN 2 and H ARTMANN 6 functionswith additional dummy dimensions that have no influence on the function value.
We use the 388D
S VM benchmark and the 124D soft-constraint version of the M OPTA 08 benchmark proposed in [17].
We set a budget of 1000 evaluations for M OPTA 08, B RANIN 2, and H ARTMANN 6 and of 2000evaluations for the other benchmarks.
Moreover, we stopped for B RANIN 2 and H ARTMANN 6 whenthe simple regret dropped below .
We also tested the algorithms on the 300D L ASSO -H IGHand the 1000D L ASSO -H
ARD benchmarks from L ASSO B ENCH [53].
These benchmarks have aneffective dimensionality of 5% of the input dimensionality, i.e., the L ASSO -H IGH and L ASSOH ARD benchmarks have 15 and 50 effective dimensions, respectively.
To study the robustness toobservational noise, we tested on noisy variants of L ASSO -H ARD and L ASSO -H IGH .
4.1 Experimental results
We begin with the six noise-free benchmarks.
Fig.
3 summarizes the performances.
On M OPTA 08,a
124D vehicle design problem
, SAASBO makes slightly faster initial progress than BA X US.
Wesuspect that this benchmark has high effective dimensionality, such that BA X US first needs to adaptthe target dimensionality to make further progress.
On the 388D S VM benchmark, BA X US adapts tothe appropriate target dimensionality where it can reach good function values faster than T U RBOand CMA-ES.
For this benchmark, Eriksson and Jankowiak [17] reported that SAASBO learned three effective dimensions.
However, the fact that A LEBO and H E SBO seem to stagnate after a fewhundred evaluations, while BA X US, T U RBO, and CMA-ES find better solutions, indicates thatoptimizing more of the 385 kernel length scales of the S VM benchmark allows for better solutions.
On the 500D H ARTMANN 6, SAASBO performs best, closely followed by BA X US.
A LEBO and H E SBO are competitive initially but converge to suboptimal solutions.
H E SBO, BA X US, SAASBO, A LEBO all find excellent solutions on the B RANIN benchmark, with the latter algorithms convergingfaster.
Next, we examine the performances on the 1000D L ASSO -H ARD and the 300D L ASSO -H
IGH thatexhibit active subspaces, here without observational noise.
BA X US achieves considerably bettersolutions than all state-of-the-art methods.
We also note that T U RBO and CMA-ES perform betterthan SAASBO and A LEBO .
While one may expect BA X US to outperform T U RBO and CMA-ESon these tasks with high input dimensions, it is surprising that SAASBO, H E SBO, and A LEBO arenot able to benefit from the present active subspace.
Here BA X US’s strategy to adaptive expand thenested subspace is superior.
Another crucial observation is that performances of BA X US vary onlyslightly across runs.
Thus, BA X US is robust despite the stochastic construction of the embedding.
Across the broad collection of benchmarks, BA X US is the only method to achieve constantly highperformance.
Noisy benchmarks.
We evaluate the algorithms also for tasks with observational noise.
Fig.
4summarizes the results.
We observe that BA X US achieves considerably better solutions for anynumber of observations than the competitors.
Moreover, we note that the performances of SAASBO,
CMA-ES, and H E SBO ( d = 20 ) degrade considerably on the L ASSO -H IGH task compared to thenoise-free formulation of the task studied above.
BA X US’s performance is equally strong as for thenoise-free case and keeps making progress after 1000 observations.
BA X US embedding ablation study.
To investigate whether the proposed family of nested randomsubspaces contributes to the superior performance of BA X US, we replaced the new embedding with asimilar family of nested H E SBO embeddings.
The results show that the proposed embedding providesa significant performance gain.
Due to space constraints, the results were moved to Appendix A.
Discussion
High-dimensional Bayesian optimization is aspiring to unlock impactful applications broadly inscience and industry.
However, state-of-the-art methods suffer from limited scalability or, in somecases, require practitioners to ‘guess’ certain hyperparameters that critically impact the performance.
This paper proposes BA X US that works out-of-the-box and achieves considerably better performancefor high-dimensional problems, as the comprehensive evaluation shows.
A key idea is to scale upthe dimensionality of the target subspace that the algorithm optimizes over.
We apply a simplestrategy that we find to work well across the board.
However, we expect substantial headroom intailoring this strategy to specific applications, either using domain expertise or by a more sophisticateddata-driven approach that, for example, learns a suitable target space.
Moreover, future work willexplore extending BA X US to structured domains, in particular the combinatorial spaces common inmaterials sciences and drug discovery.
Societal impact.
Bayesian optimization has recently become a popular tool for tasks in drugdiscovery [46], chemical engineering [8, 24, 27, 52, 54], materials science [20, 25, 26, 47, 57, 61],aerospace engineering [1, 35, 39], robotics [9, 10, 37, 41, 49], and many more.
This speaks to theprogress that Bayesian optimization has made in becoming a robust and reliable ‘off-the-shelf solver.’
However, this promise is not yet fulfilled for the newer high-dimensional Bayesian optimization fieldthat allows optimization over hundreds of ‘tunable levers.’
The applications mentioned above benefitfrom incorporating more such levers in the optimization: it allows for more detailed modeling ofan aerospace design or a more granular control of a chemical reaction, to give some examples.
Theevaluation shows that the performance of state-of-the-art methods degenerates drastically for suchhigh dimensions if the application does not meet specific requirements.
Adding insult to the injury,such requirements like the dimensionality of an active subspace cannot be determined beforehand.
The proposed algorithm achieves a robust performance over a broad collection of tasks and thus willbecome a ‘goto’ optimizer for practitioners in other fields.
Therefore, we release the BA X US code.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes] See Section 5.
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] SeeSection 5.(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes]
(b) Did you include complete proofs of all theoretical results?
[Yes] See Appendix C.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters,
how they
were chosen)?
[Yes]
(c) Did you report error bars (e.g., with respect to the random seed after running experi-
ments multiple times)?
[Yes]
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] See Appendix F.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] See Appendix F.
(b) Did you mention the license of the assets?
[Yes] See Appendix F.
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]
A Ablation study for the BA X US embedding
We conduct an ablation study to investigate the difference between the BA X US and H E SBOembeddings.
We run T U RBO, a third-party optimizer by [19], in an embedded subspace with thetwo different embeddings.
We use a version of A CKLEY 10 (ten effective dimensions, i.e., d e = 10 ),where we shift the optimum away from the origin with a random vector δ ∈ [ − 32 .
768] d ewith δ i ∼ U ( − 32 .
The function we optimize is thenf ShiftedAckley
( x ) =
f Ackley10 ( x + δ ) .
We adjust the boundaries of the search space such that f ShiftedAckley10 is evaluated on the domain
X = [ − 32 .
768] d e .
The reason for shifting the optimum is that the original A CKLEY functionhas its optimum at the origin.
In that case, any count-sketch type embedding can represent thisoptimum, even if all the important input dimensions are mapped to the same target dimension.
We add 10 dummy dimensions, such that D = 30 and set d = 20 .
With this problem-setting, the
BA X US and H E SBO embeddings have a probability of approximately 0.27 and 0.07 to contain theoptimum, respectively.
The BA X US embedding has a higher probability
The left side of Figure 5 shows the incumbent mean for T U RBO in the two different embeddedsubspaces.
The shaded regions show one standard error.
T U RBO in an BA X US embedding hassignificantly better optimization performance than in a H E SBO embedding.
The right side of Figure 5shows the distributions of the final incumbents and their median.
The BA X US embedding has asignificantly lower median and only rarely a similarly bad embedding as the H E SBO method.
We perform a two-sided Wilcoxon rank-sum statistical test to check the difference between the bestobserved function values for the two embeddings.
The difference is significant with p ≈ 0 .
The performance difference between the two embeddings depends on the characteristics of thefunction and the different dimensionalities, the input dimensionality D , the target dimensionality d ,and the effective dimensionality d e .
For problems with a few effective dimensions and many inputdimensions, the BA X US and H E SBO embeddings become more similar (see Figure 2).
However, by
Corollary 2, the BA X US embedding is, in expectation and in terms of success probability, alwaysbetter than the H E SBO embedding for arbitrary sparse axis-aligned functions.
For functions having the optimum at the origin, both embeddings contain that optimum regardless ofd : Even if all important input dimensions are mapped to the same target dimension, the optimum inthe input space can be reached by “setting” this particular target dimension to zero.
Distribution of important input dimensions.
The success probability studied in Section 3 andabove only considers the case where all the important input dimensions are mapped to differenttarget dimensions.
It might not be necessary to map all the important input dimensions to differenttarget dimensions, but it is generally beneficial if they spread out.
For example on M OPTA 08 or
S VM in Figure 3 (top left, top center), BA X US makes initial progress, then seems to be in a too lowembedding before the target dimensionality is increased and BA X US makes further progress.
If onlythe case where all d e important input dimensions are mapped to individual target dimensions allowedto make progress, BA X US would not make any progress until it reaches a suitable embedding.
Even though it is often possible to make progress without mapping all important input dimensions toindividual target dimensions, it is undoubtedly beneficial if, for a given target dimensionality, theymap to more individual target dimensions; The more the important input dimension spread out, themore the effective subspace can be covered.
Figure 6 shows a histogram of the number of individual target dimensions the d e = 10 importantinput dimensions map to in the example from above for the BA X US and H E SBO embeddings.
The bars on the very right approximately give the success probability for the H E SBO and BA X USembeddings - they correspond to a success (see Definition 2).
The BA X US embedding maps the 10important input dimensions to 9 or 10 individual target dimensions in around 74% of the cases; in the
H E SBO embedding, this happens in less than half as many cases ( 22% ).
T U RBO with BA X US embedding vs. BA X US.
We compare the simple idea of running T U RBOin a BA X US embedding with BA X US.
We run this simple approach for 11 different target dimensionalities d ( 2 , 10 , 20 , 30 , 40 , 50 , 60 , 70 , 80 , 90 , 100 ) on the L ASSO -H
ARD benchmark and showthe results with a sequential color map in Figure 7.
Only the first d = 2 -dimensional embeddingachieves the same initial speedup as BA X US, which is expected as BA X US starts in a similarlylow-dimensional initial embedding.
However, the fixed embedding cannot explore the input spacesufficiently and has the worst final solution.
High-dimensional fixed embeddings have more freedomin exploring the input space; however, they suffer from slower initial optimization performance.
BA X US has the same initial speedup as the two-dimensional fixed embedding but can explore thespace further by increasing the dimensionality of its embedding.
B Additional Lasso benchmark
In addition to the synthetic L ASSO -H IGH and L ASSO -H
ARD benchmarks studied in Section 4, weevaluate BA X US on the L ASSO -DNA benchmark from L ASSO B ENCH [53].
The L ASSO -DNAbenchmark is a biomedical classification task, taking binarized DNA sequences as input [53].
Figure 8 shows the mean performance of BA X US on the L ASSO -DNA.
Each line shows the incumbent mean; the shaded regions around the lines show one standard error.
We see the same qualitative behavior as discussed in Section 4: BA X US reaches a good initial solution faster than any other method and converges quickly.
C Theoretical foundation for the BA X US embedding
Definition 3 ( Optima-preserving count-sketch type embedding ) .
An embedding of the countsketch type is optima-preserving if each target dimension contains at most one important inputdimension.
Definition 4 ( Sparse axis-aligned function ) .
A function f : X → R with X = [ − 1 , 1] D iscalled sparse , if there exist an effective subspace Y ⊆
R d e with d e < D and a projection matrix S ⊺ ∈ R d e × D , such that for any x ∈ X there exists a y ∈ Y so that f ( x ) =
f ( S ⊺ y ) .
It is furthercalled axis-aligned , if S ⊺ is of count-sketch type and Y =
[ − 1 , 1] d e .
C.1 Proof of Lemma
Lemma 1 ( Success probability of the BA X US embedding ) .
Let D be the input dimensionality andd ≥ d e the dimensionality of the embedding.
Let β small = ⌊ Dd ⌋ and β large =
Dd ⌉ be the small andlarge bin sizes.
Then, the probability of Y ∗ (see Definition 2) with the BA X US embedding is
[Equation]
Proof.
The assignment of input dimensions to target dimensions and the signs of the input dimensionsfully define the H E SBO and BA X US embeddings.
Note that the signs do not have any effect onp A ( Y ∗ ; D, d, d e ) because they only correspond to “flipping” the input dimension in the target space.
If and only if an assignment is optima-preserving, it is possible to find a point in Y that maps to anoptimum in X for any f .
The “only if” is true because f is assumed to be sparse and axis -alignedwith d e effective dimensions.
This means that the optima in X only change their function valuesalong the d e effective dimensions.
Suppose it is possible to find a point in Y that maps to an arbitraryoptimum in X .
In that case, the assignment has to be optima-preserving because it has to be ableto adjust all the d e effective dimensions in X individually.
However, this requires each importantinput dimension to be mapped to a different target dimension.
Otherwise, there would be at least twoimportant input dimensions that cannot be changed independently.
Therefore, the probability of Y ∗is equal to the probability of an optima-preserving assignment.
As all assignments are equally likely, the probability of an assignment to be optima-preservingis equal to the number of possible optima-preserving assignments divided by the total number ofassignments.
Let us first assume that β small
= β large := β , i.e., all target dimensions have the same number ofinput dimensions and d divides D .
There are (cid:0) dd e (cid:1) ways of assigning d e important dimensions to ddifferent target dimensions.
Given one effective dimension, there are β ways in which this dimensioncan map to the target dimension.
Therefore, for the case of homogeneous bin sizes, the successprobability is given by
[Equation]
Next, we generalize Eq.
(6) for cases where d does not divide D .
Then, one must consider all possiblesplits of target dimensions between small and large numbers of contributing input dimensions.
Thisgives Eq.
It is easy to see that Eq.
(5) is equivalent to the uniform formulation in Eq.
Consider the factthat (cid:0) 00 (cid:1) =
1 and (cid:0)
x (cid:1) = 0 for all x ∈ N ++ .
The quantities d (1 + β small ) − D and D − dβ smallcan be interpreted as the number of target bins with fewer and more contributing input dimensions,respectively.
Then, for the uniform case β small = β large = β , d (1 + β small ) − D =
d , and D − dβ small = 0 .
Therefore, the success probability for the almost-uniform case is given by
[Equation]
where the last equality is true because the sum is zero unless i = d e .
C.2 Proof of Corollary 1
Corollary 1.
In the limit, the success probability of the BA X US embedding is
[Equation]
and hence matches H E SBO’s success probability
p H ( Y ∗ ; d, d e ) .
Proof.
The H E SBO embedding is constructed by choosing a target dimension and a sign uniformlyat random for each input dimension.
With this construction, the H E SBO embedding is independentof D .
[36] give an expression for the H E SBO success probability:
[Equation]
The validity of this expression can be seen by considering the probability that d e important inputdimensions are mapped to d target dimensions without collision if d ≥ d e :
[Equation]
Clearly, as D → ∞ , the H E SBO embedding has uniform bin sizes; intuitively, the H E SBOembedding and our formulation are equivalent in the limit.
We consider lim β →∞ , which is equivalent to lim D →∞ as β =
Dd and d is fixed.
We further use theuniform formulation in Eq.
(6) as a starting point because all bins are of infinite size in the limit.
[Equation] − d e )! [Equation] [Equation]
(11) = lim [Equation] d !
[Equation]
Stirling
[Equation] [Equation] [Equation] lim
[Equation]
[Equation]
lim
[Equation]
(cid:124) (cid:123)(cid:122) (cid:125)
[Equation]
βd (cid:124) (cid:123)(cid:122) (cid:125)
[Equation]
[Equation]
(cid:124) (cid:123)(cid:122) (cid:125)
[Equation]
dde (16) =
d ! ( d − d e )!
d d e (17) Eq. (17) is the failure probability of the H E SBO embedding [36].
C.3 Proof of Corollary
Corollary 2.
With the same input, target, and effective dimensionalities ( D , d , and d e ), no countsketch type embedding has a higher success probability than the BA X US embedding.
Proof.
By Definition 1, an embedding matrix S ∈ { 0 , ± 1 } D ×
d is of the count-sketch type if eachrow in S has precisely one non-zero entry.
Such an embedding can always be interpreted as disjointsets of signed input dimensions that are assigned to different target dimensions:
For the i -th inputdimension, find the column where the non-zero entry lies.
The number of the respective column givesthe target dimension; the entry in the matrix itself gives the sign.
Conversely, each target dimensionhas a set of contributing input dimensions, and we call the set of input dimensions mapping to a targetdimension a “bin”.
The sign of the input dimensions does not influence the success probability as itdoes not influence the ability of an embedding to contain the optimum.
We will prove that the BA X US embedding is optimal because every other embedding of the countsketch type has a success probability that is lower or equal to the success probability of the BA X USembedding.
We start the proof by giving the success probability for arbitrary bin sizes.
Let {+i 
+}
= 0 . Let β p be the bin size of the p -th bin.
The success probability for arbitrary bin sizes isgiven by
[Equation]
with β i > 0 , β d = D − (cid:80) d − 1 n =1 β n , and d ≥ d e .
Eq.
(18) considers all assignments of the d e effective dimensions to the d different bins. The indicatorfunction considers only cases where the important input dimensions mapped to the bins sum up to d e .
Next, if the n -th bin has one or more contributing input dimensions, we consider all the locations β nin which the important input dimension can lie. Otherwise, the β -term of this bin is always 1, theneutral element of multiplication.
Eq.
(18) is the general failure probability of any embedding of the count-sketch type as the successprobability only depends on the assignment of important input dimensions to target dimensions.
We now prove that any count sketch embedding has a success probability that is less or equal to thesuccess probability of the BA X US embedding.
[Equation]
where β = D/d .
The first term in the nominator of Eq.
(18) is equal to (cid:0) dd e (cid:1) because it counts allcases in which d e important input dimensions are assigned to d different bins in an arbitrary manner. For the inequality (cid:81) dn =1 β i n n ≤ β d e , note that (cid:80) dn =1 β n = D . Then, this inequality holds by theinequality of arithmetic and geometric means (AM-GM inequality):
[Equation]
a
[Equation]
0 , with equality if and only if a i = a j for i, j ∈ { 1 , . . . , n } .
Finally, observe that (cid:81) dn =1 β i n n is a convex function because all factors in the product are ∈ N ++ .
Therefore, the more the bin size deviates from D/d , the lower the failure probability (and equalityonly holds if all bin sizes are equal by AM-GM). By construction, the BA X US embedding is as close to homogeneous bin sizes as possible. Itcorresponds to the RHS of Eq.
(19) if d divides D ; otherwise, it is as close as possible to it as thebin sizes differ by at most 1 .
Therefore, the BA X US embedding is optimal in terms of successprobability.
D Global convergence of BA X US
We will now prove the global convergence of BA X US.
The proof idea is similar to Erikssonand Poloczek [18] but relaxes the assumption of a unique global minimizer.
Since, based on our assumptions, f is sparse and axis-aligned (see Definition 4), i.e., there exists a set of dimensions of fthat do not influence the function value whatsoever, it is crucial to make this relaxation.
BA X USdoes not have to converge to a particular configuration of these dimensions — they can take anyarbitrary value.
Theorem 2 ( BA X US convergence ) .
Under the following assumptions:
The initial points for BA X US are chosen such that ∀ δ > 0 and x ∈ [ − 1 , 1] D , ∃ ν ( x , δ ) > 0 ,such that the probability that at least one point in { x i } ends up in a ball centered at x withradius δ is at least ν
( x , δ ) ;2.
The objective f is bounded in X ,
i
e., | f
( x ) | < C , where x ∈ X and C ∈ R ++ ;3.
BA X US considers any sampled point an improvement only if it improves the current bestsolution by at least some constant γ ∈ R ++ ;4.
D is finite;5.
At least one of the minima of f is not singularly discontinuous;
BA X US converges to a global minimizer x ∗ ∈ arg min x ∈X f ( x ) for a noiseless function f .
Proof.
By Assumption (4), the fail tolerance is always finite since it is always bounded by the targetdimension ( ∀ i τ i fail ≤ d i ) which is at most equal to D ( ∀ i d i ≤ D ).
By Assumption (3), we needto make a progress of at least γ to consider a new point an improvement.
By this construction andthe fact that f is bounded (Assumption (2)), BA X US increases the target dimensionality in a finitenumber of function evaluations and eventually reaches the input dimensionality D .
Once BA X US reaches D , it behaves like T U RBO [19].
Eriksson and Poloczek [18] proved globalconvergence for T U RBO under the assumption of the existence of only one global minimizer.
However, by the convergence theorem for global search by Solis and Wets [56], page 20, BA X USconverges to some global minimizer that lies in a continuous region with non-zero probability(Assumption (5)).
E The nested family of random embeddings
We describe the method for increasing the target dimensionality under the retention of the observations.
Suppose that we have collected n observations and are in target dimension d when Algorithm 2 isinvoked.
Algorithm 2 loops over the target dimensions 1 , . . .
, d .
For each target dimension, thecontributing input dimensions are re-assigned to new bins.
If the number of contributing inputdimensions is less than b + 1 ( b : number of new bins), then it is not possible to re-assign thecontributing input dimensions to b + 1 bins.
Therefore we re-assign the contributing input dimensionsto ˆ b = min( b, l s − 1) bins, where l s is the number of contributing input dimensions to the s -th targetdimension.
This also ensures that the target dimension never grows larger than D in the BA X USembedding.
We evenly distribute the l s contributing input dimensions across the ˆ b bins by againusing the BA X US embedding.
This gives a smaller projection matrix ˜ S ⊺ which we finally use toupdate S ⊺ :
Input: transposed embedding matrix S ⊺ , number of new bins per latent dimension b , observed points
Y ∈ [ − 1 , 1] n × d .
Output: updated transposed embedding matrix S ⊺ and updated observation matrix Y for s ∈ { 1 , . . .
, d } do D s ← contributing input dimensions of s -th latent dimension. l s ← | D s | ˆ b ← min( b, l s − 1) ▷ If l s − 1 < b , we can at most create l s − 1 new bins.
Copy and append s -th column of Y ˆ b times at the end of Y . Add ˆ b zero columns at the end of S ⊺ .
σ ← signs of dimensions ∈ D s . ˜ S ⊺ ← BAxUS-Embedding ( l s , ˆ b + 1 )
▷ Re-assign input dims. homogen., ˜ S ⊺ ∈ { 0 , ± 1 } l s × ˆ b + for i ∈ { 1 , . . . , l s } , j ∈ { 1 , . . . , ˆ b + 1 } do if
{+˜ S ⊺ ij ̸ = 
 then if j > 1 then Move values that new bins to end of S ⊺ . S ⊺ D si , ˆ d − ˆ b − 1+ j ← σ i columns+}
S
S ⊺ D si ,s ←
Set zero.
Return S ⊺ and Y .
F Additional implementation details
We benchmark against SAASBO, T U RBO, H E SBO, A LEBO , and CMA-ES:•
For
SAASBO we use the implementation from ( https://github.com/
martinjankowiak/saasbo , license: none, last accessed: 05/09/2022).
• For T U RBO, we use the implementation from [19] ( https://github.com/uber-research/ TuRBO , license: Uber , last accessed: 05/09/2022).
• For H E SBO and A LEBO , we use the implementation from [36] ( https://github.com/facebookresearch/alebo , license: CC BY-NC 4.0 , last accessed: 05/09/2022).•
For the L ASSO benchmarks, we use the implementation from [53] ( https://github.com/ksehic/LassoBench , license: MIT and BSD-3-Clause , last accessed: 05/09/2022). We use GPyTorch to train the GP with the following setup: We place a top-hat prior one the Gaussianlikelihood noise, the signal variance, and the length scales of the ARD kernel.
The interval for thenoise is [0 . 005 , 0 . 2] , for the signal variance [0 . 05 , 20] , and for the lengthscales [0 .
We evaluate on the synthetic B RANIN 2 1 and H ARTMANN 6 2 functions.
Since we augment the functionwith dummy dimensions, we use the same domain for x 1 and x 2 , namely [ − 5 , 15] D for B RANIN 2and [0 , 1] D for H ARTMANN 6. Similar to T U RBO, we sample a min(100 d n , 5000) -element
Sobol sequence on which we minimizethe posterior sample.
To maximize the marginal log-likelihood of the GP, we sample 100 initialhyperparameter configurations.
The ten best samples are further optimized using the A DAM optimizerfor 50 steps.
We ran the experiments for approximately 7 , 000 core hours on Intel Xeon Gold 6130 CPUs providedby a compute cluster.