Increasing the Scope as You Learn: Adaptive Bayesian Optimization in Nested Subspaces
Abstract
Recent advances have extended the scope of Bayesian optimization (BO) to expensive-to-evaluate black-box functions with dozens of dimensions, aspiring to unlock impactful applications, for example, in the life sciences, neural architecture search, and robotics.
However, a closer examination reveals that the state-ofthe-art methods for high-dimensional Bayesian optimization (HDBO) suffer from degrading performance as the number of dimensions increases or even risk failure if certain unveriﬁable assumptions are not met.
This paper proposes BA X US that leverages a novel family of nested random subspaces to adapt the space it optimizes over to the problem.
This ensures high performance while removing the risk of failure, which we assert via theoretical guarantees.
A comprehensive evaluation demonstrates that BA X US achieves better results than the state-of-the-art methods for a broad set of applications.
Introduction
The optimization of expensive-to-evaluate black-box functions where no derivative information is available has found many applications, for example in chemical engineering [11, 28, 31, 58, 60], materials science [23, 29, 30, 32, 52, 64, 68], aerospace engineering [39, 43], hyperparameter optimization [5, 33, 38, 61], neural architecture search [36, 57], vehicle design [14, 34], hardware design [19, 49], drug discovery [51], robotics [12, 13, 41, 46, 54] and the life sciences [18, 59, 65].
Here increasing the number of dimensions (or parameters) of the optimization problem usually allows for better solutions.
For example, by exposing more process parameters of a chemical reaction, we obtain a more granular control of the process; for a design task, we may optimize a larger number of design decisions jointly; in robotics, we gain access to more sophisticated control policies.
A series of breakthroughs have recently pushed the envelope of high-dimensional Bayesian optimization and facilitated a wider adoption in science and engineering.
The key challenge for further scaling is the so-called curse of dimensionality.
The complexity of the task of ﬁnding an optimum grows exponentially with the number of dimensions [7, 22].
Recently, methods that rely on local ’trust regions’ have gained popularity.
They usually achieve good performance for problems with up to a couple of dozen input parameters.
However, we observe that their performance degrades for higher-dimensional problems.
This is not surprising, given that trust regions have a smaller volume but still the full dimensionality of the problem.
Other state-of-the-art methods suppose the existence of a low-dimensional active subspace and enjoy great scalability if they ﬁnd such a space.
The caveat is that its existence is usually not known for practical applications.
Moreover, the user needs to ‘guess’ a good upper bound on its dimensionality to enjoy a good sample efﬁciency.
In this work, we propose a theoretically-founded approach for high-dimensional Bayesian optimization, BA X US ( B ayesian optimization with a daptively e x panding s u bspace s ), that reliably achieves a high performance on a comprehensive set of applications.
BA X US utilizes a family of nested embedded spaces to increase the dimensionality of the domain that it optimizes over as it collects more data.
As a byproduct, BA X US can leverage an active subspace, if it exists, without requiring the user to ’guess’ its dimensionality.
BA X US is based on a novel random linear subspace embedding that enables a more efﬁcient optimization and has strong theoretical guarantees.
We make the following contributions:
We develop BA X US that reliably achieves excellent solutions on a broad set of high-dimensional tasks, outperforming the state-of-the-art.
We present a novel family of nested random embeddings that has the following properties: a) BA X US’ embedding provides a larger worst case guarantee for containing a global optimum than the H E SBO embedding proposed by [50].
b)
The BA X US embedding is an optimal sparse embedding , as deﬁned in Def.
c)
Its probability of containing an optimum converges to the one of the H E SBO embedding as the input dimensionality D → ∞ .
We conduct a comprehensive evaluation on a representative collection of benchmarks that demonstrates that BA X US outperforms the state-of-the-art methods.
The remainder of this paper is structured as follows.
Section 2 states the problem and discusses related work.
Section 3 presents the BA X US algorithm and the corresponding embedding.
Section 4 evaluates BA X US on a variety of benchmarks.
We give concluding remarks in Section 5.
Background
The task is to ﬁnd a minimizer
[Equation]
where X = [ − 1 , +1] D .
The objective function f : X → R is an expensive-to-evaluate blackbox function.
Hence the number of function evaluations needed to ﬁnd an optimizer is crucial.
Evaluations may be subject to observational noise, i.e., f ( x i ) + ε i , with ε i ∼ N (0 , σ 2 ) .
This work focuses on scalable high-dimensional Bayesian optimization , where the input dimensionality D is in the hundreds, and the sampling budget may comprise a thousand or more function evaluations.
Linear embeddings.
A successful approach for HDBO is to assume the existence of an active subspace [17], i.e., there exist a space Z ⊆
R d e , with d e ≤ D and a function g : Z → R , such that for all x : g ( T x ) = f
( x ) where T ∈ R d e × D is a projection matrix projecting x onto Z and d e is the effective dimensionality of the problem.
In practice, both d e and Z are unknown.
REMBO (Random embedding BO) [71] and H E SBO (Hashing-enhanced subspace BO) [50] try to capture this active subspace by a randomly chosen linear subspace.
Therefore, they generate a random projection matrix S ⊺ that maps from a d -dimensional subspace Y ∈ R d with d ≪ D (the target space ) to X .
We call d the target dimensionality .
For REMBO, each entry in S ⊺ is normally distributed.
REMBO uses a heuristic to determine a hyperrectangle in Y that it optimizes over.
Note that the bounded domain may not contain a point that maps to an optimizer of f , a risk aggravated by distortions introduced by the projection.
[6, 8, 9] proposed ideas to mitigate the issue.
H E SBO’s random projection assigns one target dimension and sign ( ± 1 ) to each input dimension.
This embedding is inspired by the count-sketch algorithm [15] for estimating frequent items in data streams.
The sparse projection matrix S ⊺ is binary except for the signs, and each row has exactly one non-zero entry.
Even though this embedding avoids REMBO’s distortions, as the authors proved, it has a lower probability of containing the optimum [40].
A LEBO [40] uses a Mahalanobis kernel and imposes linear constraints on the acquisition function to avoid projecting outside of X .
Non-linear embeddings.
Several works use autoencoders to learn non-linear spaces for optimization, trading in sample efﬁciency.
Tripp et al.
[67] change the training objective of a variational autoencoder (VAE) [37] to make the target space more suitable for optimization.
They give higher weight to better-performing points when training the variational autoencoder (VAE) and show that this improves optimization.
Moriconi et al.
[47] incorporate the training of an autoencoder directly into the likelihood maximization of a Gaussian process (GP) surrogate.
The computational cost is cubic in the number of samples and the input dimension.
Lu et al.
[42] and Maus et al.
[45] use autoencoders to learn embeddings of highly structured input spaces such as kernels or molecules.
Other approaches include partial least squares [10] or sliced inverse regression [16].
High-dimensional BO in the input space.
A popular approach to make HDBO in the input space feasible is trust regions (TRs)
The T U RBO algorithm [22] optimizes over bounded
TRs instead of the global space, adapting their side lengths and the center points during the optimization process.
By restricting function evaluations to trust regions, T U RBO addressed the problem of over-exploration; see [22] for details.
Note that the TRs have full input dimensionality, which may impact T U RBO’s ability to scale to very large dimensions.
Nonetheless, T U RBO set a new stateof-the-art by scaling to dozens on input dimensions and thousands of function evaluations.
Wan et al.
[69] extended the idea of TRs to categorical and mixed spaces by using the Hamming distance to deﬁne the TR boundaries.
SAASBO [20] uses sparse priors on the GP length scales which seems particularly valuable if the active subspace is axis-aligned.
Indeed, SAASBO can outperform T U RBO on certain benchmarks [20].
The cost of inference scales cubically with the number of function evaluations; thus, SAASBO is not expected to scale beyond small sampling budgets, which is conﬁrmed by our experiments.
Another line of research relies on the assumption that the input space has an additive structure [24, 35, 48, 72].
Additive GPs rely on computationally expensive sampling methods to learn a decomposition of the input variables, which limits the scalability of such methods to problems of moderate dimensionalities and sampling budgets [22, 50].
Wang et al.
[70] combined the meta-level algorithm LA-MCTS with T U RBO to improve optimization performance by learning a hierarchical space partition.
In Sect.
4 we evaluate the performances of T U RBO, SAASBO, A LEBO , and H E SBO.
Moreover, we study the popular CMA-ES [26] and random search [4].
The BA X US algorithm
Wang et al.
[71] showed that the REMBO embedding contains the optimum in the target space with probability one if d ≥ d e
and if there are no bounds on the target and input spaces, i.e., Y = R d and X = R D .
For d < d e , it is generally impossible to represent an optimum in X for arbitrary f as S projects to a d -dimensional subspace in X .
We call the probability of a target space to contain the optimum the success probability .
For d ≥ d e , there is a positive success probability that increases with d [40, 71].
The main problem is to set d as small as possible to avoid the detrimental effects of the curse of dimensionality but as large as necessary to achieve a high probability of Y to contain an optimum.
In reality, the active subspace and its dimensionality are usually unknown.
The performance of methods, such as REMBO [71], H E SBO [50], and A LEBO [40] depends on choosing d such that the success probability is high.
Therefore, they implicitly rely on guessing the effective dimensionality d e appropriately.
We argue that choosing the target dimensionality is problematic in many practical applications.
If chosen too small, the subspace cannot represent f sufﬁciently well.
If it is chosen too large, the curse of dimensionality slows down the optimization.
The proposed algorithm, BA X US, operates on target spaces of increasing dimensionality while preserving previous observations.
Let d init be the initial target dimensionality and m the total evaluation budget.
BA X US starts with a d init -dimensional
embedding that is increased over time until, after m D ≤ m evaluations, it roughly reaches the input dimensionality D .
With this strategy, we can leverage the efﬁciency of BO in low-dimensional spaces while guaranteeing to ﬁnd an optimum in the limit.
Increasing the target dimensionality is enabled by a novel embedding, which lets us carry over observations from previous, lower-dimensional target spaces into more high-dimensional target spaces.
We further use a TR-based approach based on Eriksson et al.
[22] to carry out optimization for high target dimensions effectively.
BA X US uses a GP surrogate [73] to model the function in the target space.
Algorithm 1 gives the pseudocode for BA X US.
In Appendix B, we prove global convergence for BA X US.
We will now present the different components in detail.
▷ Return the best observation in case of observations without noise, or the best point according to posterior mean in case of noisy observations.
S ⊺ = − +   +  − −
BBBBBB @
CCCCCCA
Input space
Before Splitting
Target space - 0.
+ 0. + 0.
xxxxxyysplit 0.
Input space
After Splitting
Target space - 0.
+ 0. + 0.
xxxxxyyy0.
c opy
3.1 The Sparse BA X US Subspace Embedding
The BA X US embedding uses a sparse projection matrix to map from Y to X .
The number of non-zero entries in this matrix is equal to the input dimensionality D .
Another embedding with this property is the H E SBO embedding [50].
Given the D and a target dimensionality d , H E SBO samples a target dimension ∈ { 1 , . . .
, d } and a sign ∈ {± 1 } for each input dimension uniformly at random.
Conversely, each target dimension has a set of signed contributing input dimensions.
We call the set of contributing input dimensions to a target dimension a bin .
These relations implicitly deﬁne the embedding matrix S ∈ { 0 , ± 1 } d × D , where each column has exactly one non-zero entry [15].
In the H E SBO embedding, the number of contributing input dimensions varies betweenand D .
The interpretation of contributing input dimensions allows for an intuitive way to reﬁne the embedding, which is shown in Figure 1.
We update the embedding matrix such that contributing input dimensions of the target dimension are re-assigned to the current bin and b new bins.
We then say that we split the corresponding target dimension.
Importantly, this type of embedding allows for retaining observations (see Figure 1).
Assume, for example, that y i is the dimension to be split.
The contributing input dimensions are re-assigned to y i and three new target dimensions y j , y k , and y l (here, b = 3 ); the observations can be retained by copying the value of the coordinate y i to the coordinates y j , y k , and y l .
Thus, the observations are contained in the old and in the new target space.
Algorithm 2 describes the procedure in detail.
In the BA X US embedding, we force each bin of a target dimension to have roughly the same number of contributing input dimensions; the bin sizes differ by at most one.
First, we create a random permutation of the input dimensions 1 , . . .
, D .
The list of input dimensions is split into min( d, D ) individual bins.
If d does not divide D , not all bins can have the same size.
We split the permutation of input dimensions such that the i -th bin has size ⌈ D/d ⌉ , if i + d ⌊ D/d ⌋ ≤ D , and ⌊ D/d ⌋ otherwise.
The ﬁrst bins have one additional element with this construction if d does not divide D .
We further randomly assign a sign to each input dimension.
The sign of the input dimensions and their assignment to target dimensions then implicitly deﬁne S ⊺ (see Figure 1).
We now show that the BA X US embedding has a strictly larger worst case success probability than the H E SBO embedding.
We establish the following two deﬁnitions.
Deﬁnition 1 ( Sparse embedding matrix ) .
We say that a matrix S ∈ { 0 , ± 1 } d × D is a sparse embedding matrix if and only if each column in S has exactly one non-zero entry.
Similarly, S ⊺ is sparse if and only if each row in S ⊺ has exactly one non-zero entry.
Deﬁnition 2 ( Success of a sparse embedding ) .
A success of a sparse embedding is the event Y ∗ =
“All d e active input dimensions are mapped to distinct target dimensions.”
It is important to note that the deﬁnition of a success is sufﬁcient but not necessary for the embedding to contain a global optimum.
For example, if the origin is a global optimum, then both embeddings contain it with probability one.
In that sense, the above deﬁnition provides a worst case guarantee .
We refer to Deﬁnition 4 in Appendix A for a formal deﬁnition of a sparse function.
In
Theorem 1, we give the success probability of the BA X US embedding.
All proofs have been deferred to Appendix A. Note that other than in the count-sketch algorithm [15], our hashing function is not pairwise independent.
However, this does not inﬂuence our theoretical analysis.
Theorem 1 ( Success probability of the BA X US embedding ) .
Let D be the input dimensionality and d ≥ d e the dimensionality of the embedding.
Let β small = (cid:4)
Dd (cid:5) and β large = (cid:6)
Dd (cid:7) be the small and large bin sizes.
Then, the probability of Y ∗ (see Deﬁnition 2) with the BA X US embedding is
[Equation]
Figure 2 shows the success probabilities of the BA X US and H E SBO embeddings for three different settings of D .
The success probability of the H E SBO embedding is given by p H ( Y ∗ ; d, d e ) =
d ! / (( d − d e )!
d d e ) (see [40] and Appendix A.3).
It is independent of D but is shown for varying d -ranges on the x -axis, therefore the visual discrepancy.
Except for when D → ∞ , the H E SBO embedding always has a non-zero failure probability, i.e., of not containing a given optimum.
The
BA X US embedding ensures that the success probability is one for d = D .
Discontinuities in the curve of the BA X US embedding occur due to the unequal bin sizes in the BA X US embedding’s success probability.
The difference between the two embeddings in Figure 2 is particularly striking when d e is high: for example, for d e = 20 H E SBO requires d = 1000 to reach a success probability of approximately 0 .
8 , whereas the BA X US embedding has a success probability of 1 as soon as d = D .
For ﬁnite D , H E SBO’s worst case success probability is smaller than BA X US’ success probability.
Corollary 1.
For D → ∞ , the success probability of the BA X US embedding is
[Equation]
and hence matches H E SBO ’s success probability
p H ( Y ∗ ; d, d e ) .
We show that the BA X US embedding is optimal among sparse embeddings.
Corollary 2.
With the same input, target, and effective dimensionalities ( D , d , and d e ), no sparse embedding has a higher success probability than the BA X US embedding.
3.2 Trust-region approach
Similar to T U RBO [22], BA X US operates in TRs.
TRs are hyper-rectangles in the input space; their shape is deﬁned by their base side length L and the GP length scales.
A TR always has volume
L , but the side length for each dimension is proportional to the corresponding length scale of the GP kernel ﬁtted to the data.
The idea of this construction is that length scales indicate how quickly the function changes along the associated dimension.
Thus, the TR is rescaled accordingly.
The volume of a TR is shrunk when T U RBO fails τ fail consecutive times to make progress, i.e., to ﬁnd a better solution.
If the algorithm consecutively makes progress for τ success = 3 times, it expands the TR.
It restarts when the base side length L of the current TR falls below a threshold L min :
= 2 − 7 .
In that case, it discards all observations for the TR and initializes a new TR on new samples.
TRs enable
T U RBO to focus on regions of the space that are close to the incumbent, i.e., the currently best solution from the algorithm’s perspective.
To choose the next evaluation point, T U RBO minimizes posterior samples of the GP on a set of candidates drawn from the TR (Thompson sampling [66]).
TRs are an essential component of BA X US as the target dimensionality usually grows exponentially during a run of the algorithm.
We use the same hyperparameter settings as T U RBO [22] with the following modiﬁcations: First, we change the criterion for when to restart a TR.
Instead of restarting a TR when it becomes too small, we increase the target dimensionality by splitting each target dimension into several new bins unless BA X US had already reached the input dimensionality:
In this case, we reset the TR base side length to the initial value.
By resetting the base side length, we also avoid convergence to a particular local minimum as the TR covers large regions of the space again.
T U RBO solves this problem by allowing for multiple parallel TRs.
Secondly, we change the fail tolerance τ fail , such that BA X US can roughly reach the input dimensionality in a ﬁxed number of evaluations.
The method for this is described in Section 3.4.
We use the same algorithmic setup as T U RBO with the aforementioned modiﬁcations.
3.3 Splitting strategy
Starting in a low-dimensional embedded space, BA X US successively grows the target dimensionality to increase the probability of containing the optimum.
By Corollary 2, it is optimal to keep the number of contributing input dimensions in the target bins as equal as possible.
At each splitting point, BA X US re-assigns the contributing input dimension of each target dimension.
Hence, the target dimensionality grows exponentially.
The number of splits required to reach some input dimensionality D is logarithmic in D .
Compared to the constant growth of the target dimensionality, BA X US has a larger evaluation budget in each split and is less dependent on the value of
D .
The algorithm starts in target dimensionality d init .
After k splits, the target dimensionality is d k = d init ( b + 1) k .
3.4 Controlling the fail tolerance
As the target dimensionality grows, the GP may need more data to model the objective function accurately.
Therefore, we want to ensure a sufﬁcient number of function evaluations before we declare a “failure”.
Recall that a failure would result in a smaller TR and eventually, after more “failures”, increase the target dimensionality.
We partition the evaluation budget into chunks of increasing sizes such that the sizes are proportional to the target dimensionalities.
Each target dimensionality then has an individual minimum evaluation budget.
Our strategy is to control the budget that BA X US has in each target dimensionality by the fail tolerance.
We choose this indirect approach because we do not want to force BA X US to increase the current target dimensionality if it makes considerable progress.
The rule we propose is based on the fail tolerance τ fail required for the increasing embedding to reach the input dimensionality D , under the assumption that BA X US always fails in making progress.
We calculate the number of splits n required to reach D by
[Equation]
with ⌊·⌉ indicating rounding to the nearest integer.
We assign each split i a split budget m si that is proportional to d i , such that P ni =1 m si = m D , where m D is the evaluation budget until D is reached:
[Equation]
m .
With that budget, we set the raw adjusted fail tolerance in the i -th split as ˜ τ
i fail =
[Equation]
, with k =
[Equation]
init k
being the number of TR shrinkages until the TR is restarted.
We aim to reduce the fail tolerance to enable BA X US to reach higher target dimensions when necessary.
Therefore, we set the fail tolerance by taking the minimum of the raw adjusted fail tolerance, and the T U RBO fail tolerances for single batches, τ i fail = max (cid:0) 1 , min (cid:0) ˜
τ i fail , d i (cid:1)(cid:1) , where d i is the target dimension after the i -th split
Setting the initial target dimension.
If the initial target dimensionality is ﬁxed, increasing the target dimensionality at an exponential rate can lead to cases where none of the target dimensionalities
d k are close to the input dimensionality D .
For example, for a 600-dimensional problem, with d init = 1 , and b = 3 , the difference between d k and D is at least 344.
Since the ﬁnal target dimensionality of BA X US is at most D , such cases are in disagreement with the exponential growth of the target dimensionality:
Either the ﬁnal target dimensionality is much less than the input dimensionality, or the last split adds only a few new dimensions.
In the ﬁrst case, we potentially reduce the success probability.
In the latter case, we assign a too-large budget to the ﬁnal split.
Therefore, we set the initial target dimensionality such that the ﬁnal target dimensionality is as close to D as possible:
[Equation]
The optimal n is given by Eq.
Alternatively, we could have set a ﬁxed initial d 0 and adjusted the growth factor b .
Experimental evaluation
In this section, we evaluate the performance of BA X US on a 388D hyperparameter optimization task, a 124D design problem, and a collection of tasks that exhibit an active subspace.
The code is available at https://github.com/LeoIV/BAxUS .
The experimental setup.
We benchmark against T U RBO [22] with one and ﬁve trust regions,
SAASBO [20], A LEBO [40], random search [4], CMA-ES [26], and H E SBO [50], using the implementations provided by the respective authors with their settings, unless stated otherwise.
For
CMA-ES, we use the P Y C MA [27] implementation.
For H E SBO and A LEBO , we use the A X implementation [1].
To show the effect of different choices of d , we run H E SBO and A LEBO with d = 10 and d = 20 .
We observed that A LEBO and SAASBO are constrained by their high runtime and memory consumption.
The available hardware allowed up to 100 evaluations for SAASBO and 500 evaluations for A LEBO for each individual run.
Larger sampling budgets or higher target dimensions for A LEBO resulted in out-of-memory errors.
We point out that limited scalability was expected for these methods, whereas the other methods scaled to considerably larger budgets, as required for scalable BO.
We initialize each optimizer with ten initial samples and BA X US with b = 3 and m D = 1000 and run 20 repeated trials.
Plots show the mean performance with one standard error.
The benchmarks.
We evaluate the selected algorithms on six benchmarks that differ considerably in their characteristics.
Following [71], we augment the B RANIN 2 and H ARTMANN 6 functions with additional dummy dimensions that have no inﬂuence on the function value.
We use the 388D S VM benchmark and the 124D soft-constraint version of the M OPTA 08 benchmark proposed in [20].
We set a budget of 1000 evaluations for M OPTA 08, B RANIN 2, and H ARTMANN 6 and of 2000 evaluations for the other benchmarks.
Moreover, we stopped for B RANIN 2 and H ARTMANN 6 when the simple regret dropped below .
We also tested the algorithms on the 300D L ASSO -H IGH and the 1000D L ASSO -H
ARD benchmarks from L ASSO B ENCH [59].
These benchmarks have an effective dimensionality of 5% of the input dimensionality, i.e., the L ASSO -H IGH and L ASSO -H
ARD benchmarks have 15 and 50 effective dimensions, respectively.
To study the robustness to observational noise, we also tested on noisy variants of L ASSO -H ARD and L ASSO -H IGH .
4.1 Experimental results
We begin with the six noise-free benchmarks.
Fig.
3 summarizes the performances.
On M OPTA 08, a 124D vehicle design problem, SAASBO initially makes slightly faster progress than BA X US.
We suspect that this benchmark has high effective dimensionality, such that BA X US ﬁrst needs to adapt the target dimensionality to make further progress.
On the 388D S VM benchmark, BA X US adapts to the appropriate target dimensionality where it can reach good function values faster than T U RBO and CMA-ES.
For this benchmark, Eriksson and Jankowiak [20] reported that SAASBO learned three active dimensions.
Yet, the fact that A LEBO and H E SBO seem to stagnate after a few hundred evaluations, while BA X US, T U RBO, and CMA-ES ﬁnd better solutions, indicates that optimizing more of the 385 kernel length scales of the S VM benchmark allows for better solutions.
On the 500D
H ARTMANN 6, SAASBO performs best, closely followed by BA X US.
A LEBO and H E SBO are competitive initially but converge to suboptimal solutions.
H E SBO, BA X US, SAASBO, A LEBO all ﬁnd excellent solutions on the B RANIN benchmark, with the latter algorithms converging faster.
Next, we examine the performances on the 1000D L ASSO -H ARD and the 300D L ASSO -H IGH that exhibit active subspaces, here without observational noise.
BA X US achieves considerably better solutions than all state-of-the-art methods.
We also note that T U RBO and CMA-ES perform better than SAASBO and A LEBO .
While one may expect BA X US to outperform T U RBO and CMA-ES on these tasks with high input dimensions, it is surprising that SAASBO, H E SBO, and A LEBO are not able to beneﬁt from the present active subspace.
Here BA X US’s strategy to adaptive expand the nested subspace is superior.
Another crucial observation is that performances of BA X US vary only slightly across runs.
Thus, BA X US is robust despite the stochastic construction of the embedding.
Across the broad collection of benchmarks, BA X US is the only method to achieve constantly high performance.
Noisy benchmarks.
We evaluate the algorithms also for tasks with observational noise.
Fig.summarizes the results.
We observe that BA X US achieves considerably better solutions for any number of observations than the competitors.
Moreover, we note that the performances of SAASBO,
CMA-ES, and H E SBO ( d = 20 ) degrade considerably on the L ASSO -H IGH task compared to the noise-free formulation of the task studied above.
BA X US’s performance is equally strong as for the noise-free case and keeps making progress after 1000 observations.
BA X US embedding ablation study.
To investigate whether the proposed family of nested random subspaces contributes to the superior performance of BA X US, we replaced the new embedding with a similar family of nested H E SBO embeddings.
The results show that the proposed embedding provides a signiﬁcant performance gain.
Due to space constraints, the results were moved to Appendix C.1.
Discussion
High-dimensional Bayesian optimization is aspiring to unlock impactful applications broadly in science and industry.
However, state-of-the-art methods suffer from limited scalability or, in some cases, require practitioners to ‘guess’ certain hyperparameters that critically impact the performance.
This paper proposes BA X US that works out-of-the-box and achieves considerably better performance for high-dimensional problems, as the comprehensive evaluation shows.
A key idea is to scale up the dimensionality of the target subspace that the algorithm optimizes over.
We apply a simple strategy that we ﬁnd to work well across the board.
However, we expect substantial headroom in tailoring this strategy to speciﬁc applications, either using domain expertise or a more sophisticated data-driven approach that, for example, learns a suitable target space.
Moreover, future work will explore extending BA X US to structured domains, particularly the combinatorial spaces common in materials sciences and drug discovery.
Societal impact.
Bayesian optimization has recently become a popular tool for tasks in drug discovery [51], chemical engineering [11, 28, 31, 58, 60], materials science [23, 29, 30, 52, 64, 68], aerospace engineering [2, 39, 43], robotics [12, 13, 41, 46, 54], and many more.
This speaks to the progress that Bayesian optimization has made in becoming a robust and reliable ‘off-the-shelf solver.’
However, this promise is not yet fulﬁlled for the newer ﬁeld of high-dimensional Bayesian optimization that allows optimization over hundreds of ‘tunable levers.’
The abovementioned applications beneﬁt from incorporating more such levers in the optimization: it allows for more detailed modeling of an aerospace design or a more granular control of a chemical reaction, to give some examples.
The evaluation shows that the performance of state-of-the-art methods degenerates drastically for such high dimensions if the application does not meet speciﬁc requirements.
Adding insult to the injury, such requirements as the dimensionality of an active subspace cannot be determined beforehand.
The proposed algorithm achieves a robust performance over a broad collection of tasks and thus will become a ‘goto’ optimizer for practitioners in other ﬁelds.
Therefore, we release the BA X US code.
Acknowledgments and Disclosure of Funding
Luigi Nardi was supported in part by afﬁliate members and other supporters of the Stanford DAWN project Ant Financial, Facebook, Google, Intel, Microsoft, NEC, SAP, Teradata, and VMware.
Leonard Papenmeier and Luigi Nardi were partially supported by the Wallenberg AI, Autonomous
Systems and Software Program (WASP) funded by the Knut and Alice Wallenberg Foundation.
Luigi Nardi was partially supported by the Wallenberg Launch Pad (WALP) grant Dnr 2021.0348.
The computations were also enabled by resources provided by the Swedish National Infrastructure for Computing (SNIC) at LUNARC, partially funded by the Swedish Research Council through grant agreement no. 2018-05973.
We would like to thank Erik Hellsten from Lund University and
Eddy de Weerd from the University of Twente for valuable discussions.
A Theoretical foundation for
the BA
X
US
embedding
For convenience, we restate Deﬁnition 1 from Section 3.
Deﬁnition 1 ( Sparse embedding matrix ) .
We say that a matrix S ∈ { 0 , ± 1 } d × D is a sparse embedding matrix if and only if each column in S has exactly one non-zero entry.
Similarly, S ⊺ is sparse if and only if each row in S ⊺ has exactly one non-zero entry.
We state the following two deﬁnitions.
Deﬁnition 3 ( Optima-preserving sparse embedding ) .
A sparse embedding matrix is optimapreserving if each target dimension (i.e., each column in S ) contains at most one active input dimension.
Deﬁnition 4 ( Sparse function / function with an active subspace. ) .
Let X =
[ − 1 , 1] D .
A function f : X → R has an active subspace , if there exist a subspace (i.e., a space Z ⊆ R d e , with d e ≤ D where d e ∈ N ++ is the effective dimensionality and N ++ = N \ { 0 } ) and a projection matrix
S ⊺ ∈ R D × d e , such that for any x ∈ X there exists a z ∈ Z so that f ( x ) =
f ( S ⊺ z ) and d e is the smallest integer with this property.
The function is called sparse if it has an active subspace and S ⊺ is a sparse embedding matrix and Z =
[ − 1 , 1] d e .
A.1 Proof of Theorem
We prove the success probability for the BA X US embedding.
For this, we ﬁrst re-state Deﬁnitionfrom Section 3.
Deﬁnition 2 ( Success of a sparse embedding ) .
A success of a sparse embedding is the event Y ∗ =
“All d e active input dimensions are mapped to distinct target dimensions.”
Theorem 1 ( Success probability of the BA X US embedding ) .
Let D be the input dimensionality and d ≥ d e the dimensionality of the embedding.
Let β small = (cid:4)
Dd (cid:5) and β large = (cid:6)
Dd
[Equation]
be the small and large bin sizes.
Then, the probability of Y ∗ (see Deﬁnition 2) with the BA X US embedding is
[Equation]
Proof.
The assignment of input dimensions to target dimensions and the signs of the input dimensions fully deﬁne the H E SBO and BA X US embeddings.
Note that the signs do not affect p A ( Y ∗ ; D, d, d e ) because they only correspond to “ﬂipping” the input dimension in the target space, and our construction ensures that the value ranges are symmetric to the origin.
An assignment is optima-preserving if and only if it is possible to ﬁnd a point in Y that maps to an optimum in X for any f .
The “only if” is true because f is assumed to be sparse with an active subspace with d e active dimensions.
This means that the optima in X only change their function values along the d e active dimensions.
Suppose it is possible to ﬁnd a point in Y that maps to an arbitrary optimum in X .
In that case, the assignment has to be optima-preserving because it has to be able to adjust all the d e active dimensions in X individually.
However, this generally requires each active input dimension to be mapped to a different target dimension (note that we require being able to represent the optimum for any f ).
Otherwise, there would be at least two active input dimensions that cannot be changed independently.
Therefore, the probability of Y ∗ equals the probability of an optima-preserving assignment.
As all assignments are equally likely under the construction, the probability of an assignment being optima-preserving is equal to the number of possible optima-preserving assignments divided by the total number of assignments.
There are (cid:0)
[Equation]
ways of distributing the d e active dimensions across the D positions (denominator in Eq.
Let us ﬁrst assume that β small
= β large := β , i.e., all target dimensions have the same number of input dimensions and d divides D .
We refer to this case as the balanced case .
There are (cid:0) dd e (cid:1) ways of distributing the d e active dimensions across the d different target dimensions.
Given one active dimension, there are β ways in which this dimension can map to the target dimension.
Therefore,
[Figure]
We would miss two input if only
This number bins as each large bin has one additional input dimension, i.e., we need two large bins to cover the two missing input dimensions.
If we only use large bins, we have one bin with one additional (ﬁctional) input dimension.
This gives the number of small bins as we need one bin with one input dimension less to achieve a correct assignment.
Clearly, two large and one smalls bin cover the input dimensions correctly.
for the balanced case, the success probability is given by
[Equation]
Next we generalize Eq.
(5) for cases where d does not divide D .
We refer to this case as the nearbalanced case .
In that case, there are two bin sizes: β small and β large with β large =
β small
+ 1 .
There are dβ large − D small bins (i.e., bins with bin size β small ) and D − dβ small large bins: D − dβ small gives the number of input dimensions that would not be covered if all bins were small.
Since β small and β large differ by 1 , this also gives the number of bins that have to be large.
Conversely, if we only had large bins, we would cover dβ large − D too many input dimensions.
Therefore, we need two large and one small bins (see Fig. 5).
We need to consider all ways of distributing the d e active dimensions across the the dβ large − D small and D − dβ small large bins (numerator in Eq.
Note that when we assign more active dimensions to small or large bins than there are such bins, the binomial coefﬁcient becomes zero, and that case does not contribute to the sum.
For each bin containing an active dimension, there are β small (or β large ) ways this dimension can lie in the bin (compare Fig. 1).
This gives the β -factors in the numerator of Eq.
It is easy to see that Eq.
(4) is equivalent to the near-balanced formulation in Eq.
(5) when d divides
D .
Consider the fact that [Equation] (cid:1)
= 1 and (cid:0)x (cid:1) =
for all x ∈ N ++ .
Then, when d divides D , β small = β large = β , d (1 + β small )
− D = d , and D − dβ small = 0 .
Therefore, the success probability for the near-balanced case is given by
[Equation]
where the last equality is true because the sum is zero unless i = d e .
A.2 Proof of Corollary
We prove the optimality of the BA X US embedding in terms of success probability.
Corollary 2.
With the same input, target, and effective dimensionalities ( D , d , and d e ), no sparse embedding has a higher success probability than the BA X US embedding.
Proof.
By Deﬁnition 1, an embedding matrix S ∈ { 0 , ± 1 } D × d is sparse if each row in S has exactly one non-zero entry.
Such an embedding can always be interpreted as disjoint sets of signed input dimensions assigned to different target dimensions:
For the n -th input dimension, ﬁnd the column with the non-zero.
The respective column gives the target dimension; the entry in the matrix itself gives the sign.
Conversely, each target dimension has a set of contributing input dimensions, and we call the set of input dimensions mapping to a target dimension a “bin”.
The sign of the input dimensions does not inﬂuence the success probability as it does not inﬂuence the ability of an embedding to contain the optimum.
We will prove that the BA X US embedding is optimal, i.e., every other sparse embedding has a success probability that is lower or equal.
We start by giving the success probability for arbitrary bin sizes.
Let β n be the bin size of the n -th bin.
By Deﬁnition 2, a success occurs if each bin contains at most one active input dimension.
Therefore, the success probability for arbitrary bin sizes has to consider the number of cases where each bin contains at most one active input dimension, and the number of bins containing one active input dimension is equal to the number of active input dimensions d e .
In a bin of size β i , the active input dimension can lie in β i different locations.
Bins not containing an active dimension do not contribute to the success probability.
We suppose w.l.o.g. that D ≥ d ; thus, every target dimension has at least one input dimension.
For each n from 1 to d , let the value i n indicate whether the n -th bin (or target dimension) contains an active dimension ( i n = 1 ) or not ( i n = 0 ).
The indicator variable 1 ( ∑ dn =1 i n )=
d e ensures that only cases where exactly d e bins contain an active input dimension are counted.
Note that Pi [Equation]
P
[Equation]
dd
[Equation]
For each case where the d e active dimensions are assigned to d e out of d disjoint bins, the term Q dn =1 β i n n accounts for the locations in which the active dimension can lie in the n -th bin.
Other cases do not contribute to the success probability.
The exponent ensures that only bins containing an active dimension contribute to the denominator.
By this reasoning, the success probability for arbitrary bin sizes is given by
[Equation]
with β n > 0 , P dn =1 β n = D , and d ≥ d e :
As in Theorem 1, the numerator of Eq.
(6) gives all ways of assigning d e active dimensions to D input dimensions.
We now prove that any sparse embedding has a success probability that is less or equal to the success probability of the BA X US embedding.
Let β small , β large , and p A ( Y ∗ ; D, d, d e ) as in Theorem 1.
Then,
[Equation]
The fact that P d e i =0 (cid:0)
We will now prove that if d divides D , the product in the numerator of Eq.
(6)is maximized, if all the factors are the same, i.e., β = Dd .
We will then show that if d does not divide D , the integer-solution of maximal value is attained for β large − β small = 1 .
First case ( d divides D , balanced case)
We now show that the following holds for the term Q dn =1 β i n n in the numerator of Eq.
(6): Q dn =1 β i n n ≤ β d e .
The numerator in Eq.
(6) can also be written as e d e ( β 1 , . . .
, β d ) where
[Equation]
is the d e -th elementary symmetric function of β 1 , . .
β d [3].
Maclaurin’s inequality [3] states that
[Equation]
In particular,
[Equation]
Then, by taking Eq.
(7) and Eq.
(8) to the power d e and multiplying by
[Equation]
with equality if and only if β i = β j for i, j ∈ { 1 , . . . , n } [3].
Therefore, the product in the numerator if Eq.
(6) is maximized if all factors are equal.
Second case ( d does not divide D , near-balanced case)
However, if d does not divide D , then β is no integer which is not feasible in our setting.
The d e -th elementary symmetric function e d e ( β ) (see Eq.
(9)) is known to be Schur-concave if β i ≥ 0 holds for all i [44].
This condition is met by β .
We use the following deﬁnition of [44]: A function f : R d → R is called Schur-concave if x ≺ y implies f ( x ) ≥ f ( y ) .
Here, x ≺ y means that y majorizes x , i.e.,
[Equation]
for all
[Equation]
and d X
[Equation]
X
[Equation] y [Equation] where x ↓ and y ↓ are the vectors of all elements in x and y in descending order [44].
We observe that all mappings of the D input dimensions into the d bins obey P n β n = D , thus the last condition is always satisﬁed for vectors of bin sizes.
Let x be the near-balanced mapping that uses only two bin sizes β large and β small with β large − β small = 1 .
Suppose that y is a mapping that is not near-balanced.
Then the difference between its largest bin and its smallest bin is at least two.
We distinguish two cases.
First, suppose that the largest bin of y has a size of at least β large + 1 .
Then x ≺ y holds because of x ↓ 1 ≤ y ↓ 1 .
Otherwise the ordered sequence y ↓ 1 starts with β large , hence there must be a bin size of at most β large − 2 by the above assumption.
Taking into account that P n β n = D holds, there must be at least one more bin of size β large in y .
Let ℓ denote its index in y ↓ .
Then we have x ≺ y because P ℓi =1 x ↓ i ≤ P ℓi =1 y ↓ i .
Therefore, the BA X US embedding has a maximum success probability among sparse embeddings.
A.3 Proof of Corollary
Corollary 1.
For D → ∞ , the success probability of the BA X US embedding is
[Equation]
and hence matches H E SBO’s success probability
p H ( Y ∗ ; d, d e ) .
Proof.
By Corollary 1, the following holds for arbitrary 1 ≤ d e ≤ d ≤ D where d, d e , D ∈ N ++ :
[Equation]
because d ! ( d − d e )!
d de is H E SBO’s success probability and hence less or equal to the success probability of BA X US.
Furthermore, by the proof of Corollary 1,
[Equation]
since β de ( dde ) ( Dde ) is larger or equal the success probability of any sparse embedding, among which BA X US is the embedding with maximum success probability and β de ( dde ) ( Dde )
[Equation]
and only if β small = β large , i.e., d divides D .
In summary,
[Equation]
We now show that, for ﬁxed d and d e , the sequences
[Equation]
and β
[Equation]
converge to the same point as D → ∞ .
We consider lim β →∞ , which is equivalent to lim D →∞ as β =
Dd and d is ﬁxed.
Note, that we can consider β =
Dd even though it is not a valid success probability when d does not divide D , since we are only interested in bounding the true success probability.
Then, lim β →∞
[Equation]
Then, by applying Stirling’s approximation [25] to the numerator and the denominator of the last factor [Equation] d [Equation]
{z }
[Equation]
d
[Equation] βd [Equation] where the following holds for the error term of the Stirling approximation [56]:
[Equation]
Then, r ( βd − d e ) r ( βd ) → 1 for β → ∞ holds since
[Equation] and
[Equation]
which both clearly go to 1 as β → ∞ . Hence, BA X US’ success probability is bounded from below and above by sequences that converge to the same point as D → ∞ . By the squeeze theorem (e.g., [62]), it follows that
[Equation]
B Consistency of BA X US
We prove the global convergence of function values for BA X US.
The proof idea is similar to Eriksson and Poloczek [21] but relaxes the assumption of a unique global minimizer.
By construction, f is sparse (see Deﬁnition 4), i.e., there exists a set of dimensions of f that do not inﬂuence the function value.
Thus, an optimal solution stays optimal regardless of how non-active dimensions are set.
This is why we must relax the assumption of a unique global minimizer in the input space.
Instead, we assume a unique global minimizer in the active subspace z ∗ ∈
Z that can map to arbitrarily many minimizers in the input space.
Theorem 2 ( BA X US consistency ) .
With the following deﬁnitions:
D1.
{ x k } ∞ k =1 is a sequence of points of decreasing function value; D2.
x ∗ ∈ arg min x ∈X f ( x ) is a minimizer in X ;
and under the following assumptions:
A1.
D is ﬁnite; A2.
f is observed without noise; A3.
f is sparse and bounded in X , i.e., ∃ C ∈ R ++ s.t.
| f
( x ) | < C ∀ x ∈ X ; A4.
At least one of the minimizers x ∗ i lies in a continuous region with positive measure; A5.
Once BA X US reached the input dimensionality D , the initial points { x i } n init i =1 after each TR restart for BA X US are chosen such that ∀ δ ∈ R ++ and x ∈ X , ∃ ν ( x , δ ) > 0 : P ( ∃ i : || x − x i || 2 ≤ δ ) ≥ ν ( x , δ ) , i.e., the probability that at least one point in { x i } n init i = ends up in a ball centered at x with radius δ is at least ν ( x , δ ) ; f ( x k ) converges to f ( x ∗ ) with probability 1.
Proof.
We ﬁrst show that BA X US must eventually arrive at an embedding equivalent to the input space.
By Assumption A1, the fail tolerance (i.e., the number of times BA X US needs to fail in ﬁnding a better solution until the TR base length is shrunk) is always ﬁnite since it is always bounded by the target dimension
( ∀ i τ i fail ≤ d i ) which is at most equal to D ( ∀ i d i ≤ D ).
By the facts that
BA X US considers any sampled point an improvement only if it improves over the current best solution by at least some constant γ ∈ R ++
and that f is bounded (Assumption A3), BA X US can only perform a ﬁnite number of function evaluations without increasing the target dimensionality of its embedding.
Once BA X US reaches D , it behaves like T U RBO [22] for which Eriksson and Poloczek [21] proved global convergence assuming a unique global minimizer.
For the case d e < D , we notice that multiple minima in the input space occur due to non-active dimensions that do not inﬂuence the function value.
The remainder of our proof is based on the convergence theorem for global search by Solis and Wets [63], which proves convergence of function values for random search with possibly multiple minima.
By considering the sequence (
[Equation]
of points of decreasing function values where { x k } ik =1 are the observations up to the i -th function evaluation, Deﬁnition D1 is satisﬁed.
Additionally, by the fact that, at each TR restart, BA X US performs random restarts with uniform probability on X , BA X US satisﬁes the assumptions of the theorem.
The Solis and Wets [63] theorem states that for a sequence { x k } ∞ k =1 of sampling points with ε ∈ R ++ ,
[Equation]
{ t : v ( x ∈ X : f ( x ) < t ) > 0 } where R ε is the set of ε -optimal function values, α is the essential inﬁmum , and v is the Lebesgue measure.
Note that the essential inﬁmum α is equal to the minimum if the minimizer lies in a continuous region of positive measure, i.e., α = f
( x ∗ i ) .
By Assumption A4 and by letting ε → 0 , f ( x k ) converges to f ( x ∗ i ) .
C Additional empirical evaluations C.1 Ablation study for the BA X US embedding
We conduct an ablation study to investigate the difference between the BA X US and H E SBO embeddings.
We run T U RBO of Eriksson et al.
[22] in an embedded subspace with the two different embeddings.
We use a version of A CKLEY 10 (ten active dimensions, i.e., d e = 10 ), where we shift the optimum away from the origin with a uniformly random vector δ ∈ [ − 32 .
768] d e with δ i ∼ U ( − 32 .
The function we optimize is then
[Equation]
We adjust the boundaries of the search space such that f ShiftedAckley10 is evaluated on the domain
X = [ − 32 .
768] d e .
The reason for shifting the optimum is that the original A CKLEY function has its optimum at the origin.
In that case, any sparse embedding contains this optimum, even if all the active input dimensions are mapped to the same target dimension.
We add 10 dummy dimensions, such that D = 30 and set d = 20 .
With this problem-setting, the
BA X US and H E SBO embeddings have a probability of approximately 0.27 and 0.07 of containing the optimum, respectively.
The left side of Figure 6 shows the incumbent mean for T U RBO in the two different embedded subspaces.
The shaded regions show one standard error.
T U RBO in an BA X US embedding has signiﬁcantly better optimization performance than in a H E SBO embedding.
The right side of Figureshows the distributions of the ﬁnal incumbents and their median.
The BA X US embedding leads to a signiﬁcantly lower median and only rarely a similarly bad embedding as the H E SBO method when combined with T U RBO.
We perform a two-sided Wilcoxon rank-sum statistical test to check the difference between the best observed function values for the two embeddings.
The difference is signiﬁcant with p ≈ 0 .
The performance difference between the two embeddings depends on the characteristics of the function and the different dimensionalities, the input dimensionality D , the target dimensionality d , and the effective dimensionality d e .
For problems with a few active dimensions and many input dimensions, the BA X US and H E SBO embeddings become more similar (see Figure 2).
However, by Corollary 2, the BA X US embedding is, in expectation and terms of success probability, always better than the H E SBO embedding for arbitrary sparse functions.
For functions with an optimum at the origin, both embeddings contain that optimum regardless of d : Even if all active input dimensions are mapped to the same target dimension, the optimum in the input space can be reached by “setting” this particular target dimension to zero.
T U RBO with BA X US embedding vs. BA X US.
We compare the simple idea of running T U RBO in a BA X US embedding with the BA X US algorithm described in Section 3.
We run this simple approach for 11 different target dimensionalities d ( 2 , 10 , 20 , 30 , 40 , 50 , 60 , 70 , 80 , 90 , 100 ) on the
L ASSO -H ARD benchmark and show the results with a sequential color map in Figure 7.
Only the ﬁrst d = 2 -dimensional embedding achieves the same initial speedup as BA X US, which is expected as BA X US starts in a similarly low-dimensional initial embedding.
However, the ﬁxed embedding cannot explore the input space sufﬁciently and has the worst ﬁnal solution.
High-dimensional ﬁxed embeddings have more freedom in exploring the input space; however, they suffer from slower initial optimization performance.
BA X US has the same initial speedup as the two-dimensional ﬁxed embedding but can explore the space further by increasing the dimensionality of its embedding.
Summing up, we observe that BA X US achieves a better performance than T U RBO with a ﬁxed embedding dimensionality.
C.2 Evaluation on an additional Lasso benchmark
In addition to the synthetic L ASSO -H IGH and L ASSO -H
ARD benchmarks studied in Section we evaluate BA X US on the L ASSO -DNA benchmark from L ASSO B ENCH [59]. The L ASSO -DNA benchmark is a biomedical classiﬁcation task, taking binarized DNA sequences as input [59].
After a worse start T U RBO ﬁnds slightly better solutions than BA X US.
C.3 Evaluation on additional MuJoCo benchmarks
We evaluate BA X US with the same baselines as in Section 4.
We use the implementation of [70] 2 , in particular we use the Gym environments Ant , Swimmer , Half-Cheetah , Hopper , Walker 2D , and Humanoid 2D , all in version 2.
For the 6392-dimensional Humanoid benchmark, we limit the target dimensionality of BA X US to 1000 dimensions to keep the split budgets sufﬁciently large.
For the other benchmarks, we do not limit the target dimensionality.
Due to the high variance between runs, we ran all methods for 50 different runs.
We summarize the results in Fig.
We observe that BA X US obtains equal or better solutions than the competitors on four out of six benchmarks.
On the 120-dimensional Walker benchmark,
BA X US is the clear winner, followed by T U RBO and CMA-ES.
On the 888-dimensional Ant benchmark, H E SBO ﬁnds the best solutions, followed by BA X US that outperforms T U RBO and
CMA-ES.
For the 102-dimensional Half-Cheetah , T U RBO produces the best solutions, followed by CMA-ES and BA X US; here, the subspace-based approaches (A LEBO and H E SBO) give signiﬁcantly worse solutions.
For the 6392-dimensional Humanoid 2D , CMA-ES obtains the best solutions, followed by BA X US, A LEBO , and H E SBO.
D The nested family of random embeddings
We describe the method for increasing the target dimensionality under the retention of the observations.
Suppose that we have collected n observations and are in target dimension d when Algorithmis invoked.
Algorithm 2 loops over the target dimensions 1 , . . .
, d .
For each target dimension, the contributing input dimensions are randomly re-assigned to new bins of given sizes.
This can, for example, be realized by, ﬁrst, randomly permuting the list of contributing input dimensions, and, secondly, dividing the list into b + 1 chunks (bins).
If the number of contributing input dimensions is less than b + 1 (remember that b is the number of new bins), then it is not possible to re-assign the contributing input dimensions to b + 1 bins.
Therefore, we re-assign the contributing input dimensions to ˆ b = min( b, l s − 1) bins, where l s is the number of contributing input dimensions to the s -th target dimension.
This also ensures that the target dimension never grows larger than D in the
BA X US embedding.
We evenly distribute the l s contributing input dimensions across the ˆ b bins by again using the BA X US embedding.
This gives a smaller (in terms of number of rows) projection matrix ˜ S ⊺ which we ﬁnally use to update S ⊺ :
Algorithm
Input: transposed embedding matrix S ⊺ , number of new bins per latent dimension b , observed points Y ∈ [ − 1 , 1] n × d . Output: updated transposed embedding matrix S ⊺ and updated observation matrix Y for s ∈ { 1 , . . .
, d } do
D s ← contributing input dimensions of s -th latent dimension of the current embedding l s ← | D s | ˆ b ← min( b, l s − 1)
▷ If l s − 1 < b , we can at most create l s − 1 new bins.
Copy and append s -th column of Y ˆ b times at the end of Y .
Add ˆ b zero columns at the end of S ⊺ .
σ ← signs of dimensions ∈
D s .
˜ S ⊺ ← Baxus-Embedding ( l s , ˆ b + 1 )
▷ Re-assign input dims.
equally 3 , ˜ S ⊺ ∈ { 0 , ± 1 } l s × ˆ b + for i ∈ { 1 , . . .
, l s } , j ∈ { 1 , . . .
, ˆ b + 1 } do if
˜ S ⊺ ij ̸ =
then if j > 1 then ▷ Move values that fall into new bins to end of S ⊺ .
S
D si , d b j ← i ▷
: columns of S ⊺
S ⊺ D si ,s ←▷ Set value
“old” column to zero.
Return S ⊺ and Y .
E Additional details on the implementation and the empirical evaluation
We benchmark against SAASBO , T U RBO, H E SBO, A LEBO , and CMA-ES: • For SAASBO, we use the implementation from [20] ( https://github.com/ martinjankowiak/saasbo , license: none, last accessed: 05/09/2022).
• For T U RBO, we use the implementation from [22] ( https://github.com/uber-research/ TuRBO , license: Uber , last accessed: 05/09/2022).
• For H E SBO and A LEBO , we use the implementation from [40] ( https://github.com/ facebookresearch/alebo , license: CC BY-NC 4.0 , last accessed: 05/09/2022).
• For the L ASSO benchmarks, we use the implementation from [59] ( https://github.com/ ksehic/LassoBench , license: MIT and BSD-3-Clause , last accessed: 05/09/2022).
We use GPyTorch (version 1.8.1) to train the GP with the following setup: We place a top-hat prior on the Gaussian likelihood noise, the signal variance, and the length scales of the Matérn 5/
ARD kernel.
The interval for the noise is [0 . 005 , 0 . 2] , for the signal variance [0 . 05 , 20] , and for the lengthscales [0 .
We evaluate on the synthetic B RANIN 2 4 and H ARTMANN 6 5 functions.
Since we augment the function with dummy dimensions, we use the same domain for x 1 and x 2 , namely [ − 5 , 15] D for
B RANIN 2 and [0 , 1] D for H ARTMANN 6.
Similar to T U RBO, we sample a min(100 d n , 5000) -element
Sobol sequence on which we minimize the posterior sample.
To maximize the marginal log-likelihood of the GP, we sample 100 initial hyperparameter conﬁgurations.
The ten best samples are further optimized using the A DAM optimizer for 50 steps.
We ran the experiments for approximately 15 , 000 core hours on Intel Xeon Gold 6130 CPUs provided by a compute cluster.