L EARNING R EPRESENTATION FOR B AYESIAN O PTIMIZATION WITH C OLLISION - FREE R EGULARIZATION
Anonymous authors
Paper under double-blind review
A BSTRACT
Bayesian Optimization has been challenged by the large-scale and highdimensional datasets, which are common in real-world scenarios.
Recent works attempt to handle such input by applying neural networks ahead of the classical
Gaussian process to learn a (low-dimensional) latent representation.
We show that even with proper network design, such learned representation often leads to collision in the latent space: two points with signiﬁcantly different observations collide in the learned latent space, leading to degraded optimization performance.
To address this issue, we propose LOC O , an efﬁcient deep Bayesian optimization framework which employs a novel regularizer to reduce the collision in the learned latent space and encourage the mapping from the latent space to the objective value to be Lipschitz continuous.
LOC O takes in pairs of data points and penalizes those too close in the latent space compared to their target space distance.
We provide a rigorous theoretical justiﬁcation for LOC O by inspecting the regret of this dynamic-embedding-based Bayesian optimization algorithm, where the neural network is iteratively retrained with the regularizer.
Our empirical results further demonstrate the effectiveness of LOC O on several synthetic and realworld benchmark Bayesian optimization tasks.
I NTRODUCTION
Bayesian optimization is a classical sequential optimization method and is widely used in various ﬁelds in science and engineering, including recommender systems (Galuzzi et al., 2019), medical trials (Sui et al., 2018), robotic controller optimization (Berkenkamp et al., 2016), scientiﬁc experimental design (Yang et al., 2019), and hyper-parameter tuning (Snoek et al., 2012), among many others.
Many of these applications involve evaluating an expensive blackbox function; therefore, the number of queries should be minimized.
A common way to model the unknown function is via Gaussian processes (GPs) (Rasmussen and Williams, 2006), which have been extensively studied under the bandit setting, as an effective surrogate model of the unknown objective function in a broad class of blackbox function optimization problems (Srinivas et al., 2010; Djolonga et al., 2013).
A key computational challenge for learning with GPs lies in optimizing speciﬁc kernels used for modeling the covariance structures.
Such an optimization task depends on the dimension of the input space.
For high-dimensional data, it is often prohibitive to train a GP model.
Meanwhile, local kernel machines are known to suffer from the curse of dimensionality (Bengio et al., 2005), while the required number of training samples could grow exponentially with the dimensionality of the data.
Therefore, dimensionality reduction and representation learning algorithms are needed to optimize the learning process.
Recently, Gaussian process optimization has been investigated in the context of latent space models.
For example, deep kernel learning (Wilson et al., 2016) learns a (low-dimensional) data representation and a scalable kernel simultaneously via an end-to-end trainable deep neural network.
In general, the neural network is trained to learn a simpler latent representation with reduced dimension and has the structure information already embedded for the GP.
Combining the representation learned via a neural network with GP could improve the scalability and extensibility of classical
Bayesian optimization, but it also poses new challenges for the optimization task, such as dealing with the tradeoff between representation learning and function optimization (Tripp et al., 2020).
As we later demonstrate, a critical challenge brought by introducing representation learning into
Bayesian optimization is that the latent representation is prone to collisions : two points with signiﬁcantly different observations can get too close, and therefore collide in the latent space.
The collision effect in latent space models for Bayesian optimization is especially evident when information is lost during dimensionality reduction and/or when the training data is limited in size.
As illustrated in
Figure 1, when passed through the neural network, data points with drastically different observations are mapped to close positions in the latent space.
Such collisions could be regarded as additional noise introduced by the neural network.
Although Bayesian optimization is known to be robust to mild noisy observations (Bogunovic et al., 2018), the collision in latent space could be harmful to the optimization performance, as it is non-trivial to model the collision into the acquisition function explicitly.
Also, the additional noise induced by the collision effect will further loosen the regret bound for classical Bayesian optimization algorithms (Srinivas et al., 2010).
Overview of main results To mitigate the collision effect, we propose a novel regularization scheme that can be applied as a simple plugin amendment for the latent space based Bayesian optimization models.
The proposed algorithm, namely Latent
Space Optimization via Collision-free regularization (LOC O ), leverages a regularized regression loss function to optimize the latent space for Bayesian optimization periodically.
Concretely, our collision-free regularizer is encoded by a novel pairwise collision penalty function deﬁned jointly on the latent space and the output domain.
In order to mitigate the risk of collision in the latent space (and consequently boost the optimization performance), LOC O applies the regularizer to minimize the collisions uniformly in the latent space.
We further note that for Bayesian optimization tasks, collisions in regions close to the optimum are more likely to mislead the optimization algorithm.
Based on this insight, we propose an optimization-aware regularization scheme that assigns higher weight to the collision penalty on those pairs of points closer to the optimum region in the latent space.
This algorithm, which we refer to as Dynamically-Weighted LOC O , is designed to dynamically assess the importance of a collision during optimization.
Compared with the uniform collision penalty in the latent space, the dynamic weighting mechanism has demonstrated drastic improvement over the state-of-the-art latent space based Bayesian optimization models.
We summarize our key contributions as follows:
I. We investigate latent space based Bayesian optimization, and expose the limitations of existing latent space optimization approaches due to the collision effect on the latent space (Section 3).
II.
We propose a novel regularization scheme as a simple plugin amendment for latent space based Bayesian optimization models.
Our regularizer penalizes collisions in the latent space and effectively reduces the collision effect.
Furthermore, we propose an optimization-aware dynamic weighting mechanism for adjusting the collision penalty to improve the effectiveness of regularization for Bayesian optimization (Section 4).
III.
We provide theoretical analysis for the performance of Bayesian optimization on regularized latent space (Section 5).
IV.
We conducted an extensive empirical study on several synthetic and real-world datasets, including a real-world case study for cosmic experimental design, and demonstrate the promising empirical performance for our algorithm (Section 6).
R ELATED W ORK
This section provides a short survey on recent work in Bayesian learning, which was designed to overcome the high-dimensionality challenge for Gaussian process regression tasks and Bayesian optimization.
Different surrogate models with internal latent space Some alternative surrogate models have been proposed to replace classical GP in Bayesian optimization to overcome the challenge of highdimensional and highly-structured input in BO.
Deep Network for Global Optimization (DNGO)
Snoek et al.
(2015) uses a pre-trained deep neural network with a Bayesian linear regressor at the last hidden layer of the network as the surrogate model.
More generally, Deep Kernel Learning (DKL) combines the power of the Gaussian process and neural network by introducing a deep neural network g to learn a mapping g : X → Z from the input domain X to a latent space
Z (Wilson et al., 2016).
It uses the latent representation z ∈ Z as the input of the base Gaussian process.
The neural network g and a spectral mixture-based kernel k form a scalable expressive closed-form deep covariance kernel, denoted by k DK ( x i , x j ) →
k ( g ( x i ) , g ( x j )) .
The deep kernel allows end-to-end learning and Bayesian optimization on the original input space.
Representation learning and latent space optimization Instead of reducing the dimensionality and performing optimization in an end-to-end process, other methods aim to optimize in a related low-dimensional space ﬁrst and then map the solution back to the original input space.
Djolonga et al.
(2013) assume that only a subset of input dimensions varies, and the kernel is smooth (i.e. with bounded RKHS norm).
Under these assumptions, the underlying subspace is learned via low-rank matrix recovery.
Random feature is another solution under this setting (Rahimi et al., 2007; Letham et al., 2020; Binois et al., 2015;
Nayebi et al., 2019; Wang et al., 2016).
It is known that a random representation space of sufﬁciently large dimension is guaranteed to contain the optima with high probability.
Mutn ` y and Krause (2019) consider Quadrature Fourier Features (QFF)—as opposed to
Random Fourier Feature (RFF) in Rahimi et al.
(2007)—to overcome the variance starvation problem, and proved that Thompson sampling and GP-UCB achieve no-regret with squared exponential kernel in optimization tasks.
However, both RFF and QFF methods rely on a key assumption that the function to be optimized has a low effective dimension .
In contrast, as discussed in Section 6 and the supplemental materials, we show that LOC O performs well for challenging high-dimensional
BO problems where algorithms relying on the low effective dimension assumption may fail.
Another line of work on latent space optimization uses autoencoders to learn low-dimensional representations of the inputs to improve the scalability and capability to leverage the structural information (Mathieu et al., 2019), (Ding et al., 2020), (G ´ omez-Bombarelli et al., 2018;
Huang et al., 2015; Tripp et al., 2020;
Lu et al., 2018).
Mathieu et al.
, Ding et al.
(2020) focus on disentangled representation learning that breaks down, or disentangles, each feature into narrowly deﬁned variables and encodes them as separate dimensions.
Tripp et al.
(2020) iteratively train the autoencoder with a dynamic weighting scheme when performing optimization to improve the embedding.
Grifﬁths and Hern ´ andez-Lobato (2020) and Letham et al.
(2020) enforce certain properties on the representation space to improve the optimization performance.
To the best of the authors’ knowledge, collision of the embeddings has not been explicitly studied.
Binois et al.
(2015) propose a warped kernel to guarantee the injectivity in the random linear embedding, which is not applicable in neural network-based methods.
A common challenge in applying these techniques to generic optimization tasks lies in the assumption on the accessibility of training data: Bayesian optimization often assumes limited access to labeled data, while surrogate models built on deep neural networks often rely on abundant access to data for pretraining.
Another problem lies in the training objective: During the training phase, these surrogate models typically focus on improving the regression performance, and do not explicitly address the artifact caused by collisions of the learned embeddings, which—as we later demonstrate in Section 3.3—could be harmful to sequential decision-making tasks.
P ROBLEM S TATEMENT
In this section, we introduce necessary notations and formally state the problem.
We focus on the problem of sequentially optimizing a function f : X → R , where X ⊆
R d is the input domain.
At iteration t , we pick a point x t ∈ X , and observe the function value perturbed by additive noise: y t
= f ( x t ) + (cid:15) t with (cid:15) t ∼ N (0 , σ 2 ) being i.i.d.
Gaussian noise.
Our goal is to maximize the sum of rewards (cid:80)
Tt =1 f ( x t ) over T iterations, or equivalently, to minimize the cumulative regret
R T := (cid:80) Tt =1 r t , where r t := max x ∈X f ( x ) − f ( x t ) denotes the instantaneous regret of x t .
3. B AYESIAN O PTIMIZATION
Formally, we assume that the underlying function f is drawn from a Gaussian process
GP ( m ( x ) , k ( x, x (cid:48) )) , where m ( x ) is the mean function and k ( x, x (cid:48) ) is the covariance function.
At iteration t , given the selected points A t = { x 1 , . . .
, x t } and the corresponding noisy evaluations y t =
[ y 1 , . . .
, y t ] (cid:62) , the posterior over f also takes the form of a GP, with mean µ t ( x ) =
k t ( x ) (cid:62) ( K t + σ 2 I ) − 1 y t , covariance k t ( x, x (cid:48) )
=
k ( x, x (cid:48) )
− k t ( x ) (cid:62) ( K t + σ 2 I ) − 1 k t ( x (cid:48) ) , and variance σ 2 t ( x ) =
k t ( x, x ) , where k t ( x ) =
[ k ( x 1 , x ) , . . .
, k ( x t , x )] (cid:62) and K t is the positive deﬁnite kernel matrix
[ k ( x, x (cid:48) )] x,x (cid:48) ∈
A t (Rasmussen and Williams, 2005).
After obtaining the posterior, one can compute the acquisition function α : X → R , which is used to select the next point to be evaluated.
Various acquisition functions have been proposed in the literature, including popular choices such as Upper Conﬁdence Bound (UCB) (Srinivas et al., 2010) and Thompson sampling (TS) (Thompson, 1933).
Remark.
Regret is commonly used as performance metric for BO methods.
In this work we focus on the simple regret r ∗ T = max x ∈X f ( x ) − max t ≤ T f ( x t ) and cumulative regret R ( T ) =
(cid:80) Tt =1 r t .
3. L ATENT S PACE O PTIMIZATION
Recently, Latent Space Optimization (LSO) has been proposed to solve Bayesian optimization problems on complex input domains (G ´ omez-Bombarelli et al., 2018;
Huang et al., 2015; Tripp et al., 2020;
Lu et al., 2018).
LSO learns a latent space mapping g : X → Z to convert the input space X to the latent space Z .
Then, it constructs an objective mapping h :
Z → R such that f ( x ) ≈
h ( g ( x )) , ∀ z ∈ Z .
In this paper, we model the latent space mapping g as a neural network; the neural network g and the base kernel k together are regarded as a deep kernel , denote by k nn ( x, x (cid:48) )
=
k ( g ( x ) , g ( x (cid:48) ))
(Wilson et al., 2016).
In this context, the actual input space for BO is the latent space Z and the objective function is h .
With the acquisition function α nn ( x ) :
= α ( g ( x )) , we do not compute an inverse mapping g − 1 as opposed to the aforementioned autoencoder-based
LSO algorithms (e.g. Tripp et al.
(2020)), since BO directly select x t = arg max x ∈X α nn ( x ) ∀ t ≤ T .
In our analysis, we use squared exponential kernel, i.e. k SE ( x, x (cid:48) )
=
σ 2 SE exp (cid:16) − ( x − x (cid:48) ) 2 2 l (cid:17) .
3. T HE C OLLISION E FFECT OF LSO
When the mapping g : X → Z is represented by a neural network, it may cause undesirable collisions between different input points in the latent space Z .
Under the noise-free setting, we say there exists a collision in Z , if ∃ x i , x j ∈ X , such that when g ( x i ) =
g ( x j ) , | f ( x i ) − f ( x j ) | > 0 .
Such collision could be regarded as additional (unknown) noise on the observations introduced by the neural network g .
Given a representation function g , noisy observations y =
f ( x )+ (cid:15) , we say that there exists a collision, if for λ > 0 , there exist x i , x j ∈ X ,
such that | g ( x i ) − g ( x j ) | < λ | y
i − y j | .
When the distance between a pair of points ( x i , x j ) in the latent space is too close compared to their difference in the output space, the different output values
y i , y j for the collided points in the latent space could be interpreted as the effect of additional observation noise for g ( x i ) (or g ( x j ) ).
In general, collisions could degrade the performance of LSO.
Since the collision effect is a priori unknown, it is often challenging to be dealt with in LSO, even if we regard it as additional observation noise and increase the (default) noise variance in the Gaussian process.
Thus, it is necessary to mitigate the collision effect by directly restraining it in the representation learning phase.
One potential method to avoid collision could be tuning the design of neural networks.
However, we empirically show that increasing the network complexity often does not help to reduce the collision.
The study is posed in Appendix B.2.
We consider a low-noise setting where the collision can play a more signiﬁcant role in degrading the optimization performance.
As is shown in Figure 1 that the collision could result in larger difﬁculty in the optimization task.
And when a collision exists, it is hard to distinguish it from the observation.
Therefore we focused on treating the collision when deﬁning the penalty instead of dealing with the stochasticity.
L ATENT S PACE O PTIMIZATION VIA C OLLISION - FREE R EGULARIZATION
We now introduce LOC O as a novel algorithmic framework to mitigate the collision effect.
4. O VERVIEW OF THE LOC O A LGORITHM
The major challenge in restraining collisions in the latent space is that—unlike the formulation of the classical regression loss—we cannot quantify it based on a single training example.
We can, however, quantify collisions by grouping pairs of data points and inspecting their corresponding observations.
[Figure]
We deﬁne the collision penalty based on pairs of inputs and further introduce a pair loss function to characterize the collision effect.
Based on this pair loss, we propose a novel regularized latent space optimization algorithm 1 , as summarized in Algorithm 1.
The proposed algorithm concurrently feeds the pair-wise input into the same network and calculates the pair loss function.
We demonstrate this process in Figure 2.
Given a set of labeled data points, we can train the neural network to create an initial latent space representation similar to DKL (Wilson et al., 2016) 2 .
Once provided with the initial representation, we can then reﬁne the latent space by running LOC O and periodically update the latent space (i.e. updating the learned representation after collecting a batch of data points) to mitigate the collision effect as we gather more labels.
1: Input : Penalty parameter λ (cf.
Equation 1), regularization weight ρ (cf.
Equation 3), importance weight parameter ζ (cf.
Equation 2), neural network g , parameters θ t =
( θ h,t , θ g,t ) , total time
T ; α (
( x, θ ,t ))
(cid:46) acquisition function evaluation x (cid:46) update observation 5: θ t +1 ← retrain θ t with the pair loss function L ρ,λ,ζ,g ( θ t , D t ) as in Equation6: Output : max t y t
4. C OLLISION P ENALTY
This subsection aims to quantify the collision effect based on the deﬁnition proposed in Section 3.3.
As illustrated in Figure 2, we feed pairs of data points into the neural network and obtain their latent space representations.
Apart from maximizing the GP’s likelihood, we concurrently calculate the amount of collision on each pair and incur a penalty when the value is positive.
For x i , x j ∈ X , y i =
f ( x i ) + (cid:15) , y j = f ( x j ) + (cid:15) are the corresponding observations, and z i = g ( x i ) , z j = g ( x j ) are the corresponding latent space representations.
We deﬁne the collision penalty as
[Equation]
where λ is a penalty parameter that controls the smoothness of the target function h : Z → R .
As a rule of thumb, one can estimate λ by sampling from the original data distribution P ( X, Y ) , ( X, Y ) ∈ X × R , and choose the λ to be the maximum value such that (cid:80) i,j max( λ | y i − y j |−| x i − x j | , 0) =(i.e. to provide an upper bound for λ by keeping the total collision in the input domain to be zero).
4. I MPORTANCE -W EIGHTED C OLLISION -F REE R EGULARIZER
Note that it is challenging to universally reduce the collisions by minimizing the collision penalty and the GP’s regression loss—this is particularly the case with a limited amount of training data.
Fortunately, for optimization tasks, it is often unnecessary to learn ﬁne-grained representation for suboptimal regions.
Therefore, we can dedicate more training resources to improve the learned latent space pertaining to the potentially near-optimal regions.
Following this insight, we propose to use a weighted collision penalty function, which uses the objective values for each pair as an importance weight in each iteration.
Formally, for any pair (( x j , z j , y j ) , ( x i , z i , y i )) in a batch of observation pairs D t
= { (( x m , z m , y m ) , ( x n , z n , y n )) } m,n where x n , x m ∈
A t and y n , y m ∈
y t , we deﬁne the importance-weighted penalty function as [Equation] with
[Equation]
Here the importance weight ζ is used to control the aggressiveness of the weighting strategy.
Combining the kernel learning objective—negative log likelihood and the collision penalty for GP, we deﬁne the pair loss function L ρ,λ,ζ,g as
[Equation]
Here, − log( P ( y t | A t , θ t ))
= − 12 y (cid:62) t ( K t + σ 2 I ) − 1 y t − 12 | ( K t + σ 2 I ) | − t 2 log(2 π ) is the learning objective for the GP (Rasmussen and Williams, 2005).
ρ denotes the regularization weight; as we demonstrate in Section 6, we initialize the regularization weight ρ to keep the penalty at the same order of magnitude as the negative log likelihood.
An alternative training process is to minimize the regression loss and the collision penalty alternatively.
We observe in our empirical study that both training processes could lead to reasonable convergence behavior of the LOC O training loss.
D ISCUSSION AND T HEORETICAL I NSIGHT
This subsection discusses the theoretical insight underlying the collision-free regularizer, by inspecting the effect of regularization on the regret bound of LOC O where the constantly trained neural network feeds a dynamic embedding to the Gaussian process.
While the key idea for bounding the regret of UCB-based GP bandit optimization algorithms follows the analysis of Srinivas et al. (2010), two unique challenges are posed in the analysis of LOC O .
Firstly, unlike previous work in Srinivas et al. (2010), the neural network is constantly retrained along with the new observations.
Thus, the input space for the downstream Gaussian process could be highly variant.
For the discussion below, we consider a stationary and monotonic kernel, and assume that retraining the neural network g does not decrease the distance between data points in the latent space.
It is worth noting that, although not strictly enforced, such monotonicity behavior is naturally encouraged by our proposed regularization, which only penalizes the pair of too-close data in the latent space.
Under the above assumption, the internal complexity of neural network training still makes it challenging to bound the regret w.r.t the dynamics of the neural network.
Thus, we investigate the dynamics of the mutual information term in the regret bound, and justify the proposed collision-free regularizer by showing that penalizing the collisions tends to reduces the upper bound on the regret.
We ﬁrst consider a discrete decision set and then leverage the desired Lipschitz continuity on the regularized space to extend our results to the continuous setting (cf full proofs in Appendix A).
Proposition 1.
Let Z be a ﬁnite discrete set.
Let δ ∈ (0 , 1) , and deﬁne β t = 2 log (cid:0) |Z| t 2 / 6 δ (cid:1) .
Suppose that the objective function h : Z × θ −→ R deﬁned on Z and parameterized by θ is a sample from GP.
Furthermore, consider a stationary and monotonic kernel, and assume that retraining the neural network g does not decrease the distance between data points in the latent space.
Running GP-UCB with β t for a sample h of a GP with mean function zero and stationary covariance function k
( x, x (cid:48) ) , we obtain a regret bound of O ∗ ( (cid:112) log( |Z| ) T ( γ T − I
( h ( z T , θ h, 0 ); φ T )) with high probability.
More speciﬁcally, with C 1 = 8 / log (cid:0) 1 + σ − 2 (cid:1) , we have
[Equation]
Here γ T is the maximum information gain after T iterations, and φ T as the identiﬁcation of the collided data points on Z .
γ T is deﬁned as γ [Equation] =
[Equation]
The collision regularization reduced the maximum mutual information by a speciﬁc term dependent on the distribution of the noise caused by the collision of data points.
The distribution is dynamic and determined by the complex learning process of the neural network.
In the following, we show that the mutual information is bounded within a given interval:
Assume φ t is a random variable that identify z t ∈ Z , y t ∈ Y from its collided points, and the variance of the collision is σ 2 col , then we have
[Equation]
This means that if φ t is a random variable sampled from a Gaussian distribution deﬁned on h , then I ( h ( z T , θ h, 0 ); φ T ) is maximized.
The regularization also constrains the function h to be Lipschitz-continuous with a Lipschitz constant, enabling a slightly narrower regret bound.
Proposition 2.
Let Z ⊂ [0 , r ] d be compact and convex, d ∈ N, r > 0 and λ ≥ 0 .
Suppose that the objective function h : Z × θ −→ R deﬁned on Z and parameterized by θ is a sample from GP and is Lipschitz continuous with Lipschitz constant λ .
Let δ ∈ (0 , 1) , and deﬁne β t =
[Equation]
λrdt 2 (cid:1) .
Furthermore, consider a stationary and monotonic kernel, and assume that retraining the neural network g does not decrease the distance between data points in the latent space.
Running GP-UCB with β t for a sample h of a GP with mean function zero and stationary covariance function k
( x, x (cid:48) ) , we obtain a regret bound of O ∗ ( (cid:112) dT ( γ T − I ( h ( z T , θ h, 0 ); φ T )) with high probability.
More speciﬁcally, with C 1 = 8 / log (cid:0) 1 + σ − 2 (cid:1) , we have
[Equation]
Here γ T is the maximum information gain after T iterations, and φ T as the identiﬁcation of the collided data points on Z .
γ T is deﬁned as γ T :=
[Equation]
E XPERIMENTS
In this section, we empirically evaluate our algorithm on several synthetic and real-world benchmark blackbox function optimization tasks.
All experiments are conducted on Google Cloud GPU instance (4 vCPUs, 15 GB memory, Tesla T4 GPU) and Google CoLab high-RAM GPU instance.
6. E XPERIMENTAL S ETUP
We consider ﬁve baselines in our experiments.
Three popular optimization algorithms—particle swarm optimization (PSO) (Miranda, 2018), Tree-structured Parzen Estimator Approach (TPE) (Bergstra et al., 2011), a BoTorch (Balandat et al., 2020) implementation of Trust Region Bayesian Optimization (TuRBO) (Eriksson et al., 2019), and standard Bayesian optimization (BO) (Nogueira, 2014) which uses Gaussian processes as the statistical model—are tuned in each task.
Another baseline we consider is the sample-efﬁcient LSO (SE LSO) algorithm, which is implemented based on the algorithm proposed by Tripp et al.
We also compare the non-regularized latent space optimization (LSO), LOC O with uniform weights (i.e. ζ = 0 , referred to as LOC O ), and the dynamically-weighted LOC O (i.e. with ζ > 0 , referred to as DW LOC O ) proposed in this paper.
One crucial problem in practice is tuning the hyper-parameters.
The hyper-parameters for GP are tuned for periodically retraining in the optimization process by minimizing the loss function on a [Figure] validation set.
For all our tasks, we choose a simple neural network architecture due to the reasoning in section 3.3, as well as due to limited and expensive access to labeled data under the BO setting.
The coefﬁcient ρ is, in general, selected to guarantee a similar order for the collision penalty to GP loss.
The λ should be tolerant of the additive noise in the evaluation.
In practice, we choose the simple setting λ
= 1 and ﬁnd it perform well.
We also include a study of the parameter choice in the appendix.
ζ controls the aggressiveness of the importance weight.
While ζ should not be too close to zero (equivalent to uniform weight) , an extremely high value could make the regularization overly biased.
Such a severe bias could allow a heavily collided representation in most of the latent space and degrade regularization effectiveness.
The value choice is similar to the inverse of the temperature parameter of softmax in deep learning (Hinton et al., 2015).
Here we use ζ = 1 for simplicity and ﬁnd it robust to different tasks.
All experiments are conducted on the pre-collected datasets.
We defer the detailed experimental setup to Appendix C.
6. D ATASETS AND R ESULTS
We now evaluate LOC O on three synthetic datasets and three real-world datasets.
We demonstrated the improvement in LOC O that is enabled by the explicit collision mitigation in the lowerdimensional latent space in terms of average simple regret.
2D Shape Area Maximization The dSprites dataset (Matthey et al., 2017) consists of images of size 64 × 64 containing 2d Shapes with different scales, rotations, and positions.
Each pixel value of the images are binary, hence x ∈ { 0 , 1 } 64 × 64 .
The goal is to generate a shape x with maximum area, which is equivalent to ﬁnding arg max x
[Equation]
64 × 64 i x i where i corresponds to the pixel index.
The neural network is pretrained on 50 data points.
To meet the limitation of memory on our computing instance, we uniformly sample 10000 points from the original dataset and approximately maintain the original distribution of the objective value.
The DW LOC O and LOC O outperform or matches the baseline methods on this dataset.
Sum 200D We create a synthetic dataset Sum 200D of 200 dimensions.
Each dimension is independently sampled from a standard normal distribution to maximize the uncertainty on that dimensions and examine the algorithm’s capability to solve the medium-dimensional problem.
We want to maximize the label f
( x ) = (cid:80) 200 i =1 e x i which bears an additive structure and of non-linearity.
The neural network is pretrained on 100 data points.
As illustrated by ﬁgure 3b DW LOC O and LOC O could signiﬁcantly outperform baselines that do not speciﬁcally leverage the additive structures of the problem.
Rastrigin-2D The Rastrigin function is a non-convex function used as a performance test problem for optimization algorithms.
It was ﬁrst proposed by Rastrigin (1974) and used as a popular benchmark dataset for evaluating Gaussian process regression algorithms (Cully et al., 2018).
Formally, the 2D Rastrigin function is f ( x ) = 10 d + (cid:80) di
=1 x 2 i − 10 cos(2 πx i ) , d = 2 .
For convenience of comparison, we take the − f ( x ) as the objective value to make the optimization tasks a maximization task.
Supernova-3D Our ﬁrst real-world task is to perform maximum likelihood inference on three cosmological parameters, the Hubble constant H 0 ∈ (60 , 80) , the dark matter fraction Ω M ∈ (0 , 1) , and the dark energy fraction Ω A ∈ (0 , 1) .
The likelihood is given by the Robertson-Walker metric, which requires a one-dimensional numerical integration for each point in the dataset from Davis et al.
The neural network is pretrained on 100 data points.
As illustrated by ﬁgure 3e, both
LOC O and DW
LOC
O demonstrate its consistent robustness.
Among them, DW LOC O slightly outperform LOC O the early stage.
Water Converter Conﬁguration-16D
This UCI dataset we use consists of positions and absorbed power outputs of wave energy converters (WECs) from the southern coast of Sydney.
The applied converter model is a fully submerged three-tether converter called CETO.
16 WECs locations are placed and optimized in a size-constrained environment.
Redshift Distribution-14D Careful accounting of all the requirements and features of these experiments becomes increasingly necessary to achieve the goals of a given cosmic survey.
SPOKES (SPectrOscopic KEn Simulation) is an end-to-end framework that can simulate all the operations and critical decisions of a cosmic survey (Nord et al., 2016).
In this work, we use SPOKES to generate galaxies within a speciﬁed window of distances from Earth.
We then minimize the Hausdorff distance between the desired redshift distribution and the simulation of speciﬁc cosmological surveys generated by SPOKES.
In our experiments, the neural network is pretrained with 400 data points.
As illustrated by ﬁgure 3f, the simple regret of DW LOC O drops slower yet eventually outperforms or matches other baselines’ performances.
In general, our experimental results consistently demonstrate the robustness of our methods against collisions in the learned latent space.
Our method outperforms or matches the performance of the best baselines in all scenarios.
When compared to the sample-efﬁcient LSO, DW LOC O performs better in most cases and shows a steady capability to reach the optimum by explicitly mitigating the collision in the latent space.
Due to the dynamics of representation learning process, it is difﬁcult to claim that the performance improvement brought by dynamic weighting is universal.
This aligned with the observation in the experiments that DW LOC O brings observable improvement in the regret curve at a certain stage for an optimization task and achieve an ultimate performance that at least matches LOC O .
In contrast, the sample-efﬁcient LSO might fail due to the collision problem.
C ONCLUSION
We have proposed a novel regularization scheme for latent-space-based Bayesian optimization.
Our algorithm—namely LOC O —addresses the collision problem induced by dimensionality reduction and improves the performance for latent space-based optimization algorithms.
We show that the regularization effectively mitigates the collision problem in the learned latent spaces and, therefore, can boost the performance of the Bayesian optimization in the latent space.
We demonstrate solid empirical results for LOC O on several synthetic and real-world datasets.
Furthermore, we demonstrate that LOC O can deal with high-dimensional input that could be highly valuable for real-world experiment design tasks such as cosmological survey scheduling. A P ROOFS
In this section, we provide proofs for our main theoretical results (Proposition 1 and Proposition 2).
A. P ROOF OF P ROPOSITION 1: R EGRET B OUND ON D ISCRETE D ECISION S ET
We follow the proof structure in Srinivas et al.
and introduce new notations to characterize the learning process of the neural network and the collision in the proof.
Before proving Proposition 1, we ﬁrst introduce a few useful lemmas.
Lemma 1.
Pick δ ∈ (0 , 1) and set β t = 2 log( | D | π t /δ ) , where (cid:80) t ≥ 1 π − 1 t = 1 , π t > 0 .
Then with probability [Equation] [Equation]
[Equation]
Here θ g,t − 1 is the parameter for g at time step t − 1 .
θ h,t − 1 is the parameter for h at time step t − 1 .
Proof.
∀ x ∈ D, ∀ t ≥ 1 , Conditioned on y t − 1 = ( y 1 , ..., y t − 1 ) , { x 1 , ..., x t − 1 } are deterministic, and h ( g ( x, θ g,t − 1 ) , θ h,t − 1 ) ∼ N ( µ t − 1 ( g ( x, θ g,t − 1 )) , σ 2 t − 1 ( g ( x, θ g,t − 1 ))) .
Then using the subgaussianity of h , we have
[Equation]
Applying the union bound, with probability ≥ 1 − | D | e − β t / 2 , ∀ x ∈ D |
[Equation]
( g
[Equation]
x, θ
[Equation]
Let | D | e − β t / 2 = δ/π t , applying the union bound for ∀ t ∈ N the statement holds.
Lemma 2.
Consider a stationary and monotonic kernel, and assume that retraining the neural network g does not decrease the distance between data points in the latent space.
Then ∀ t ≥ 1 ,
[Equation]
Here θ g,T is the parameter for g at time step T .
Proof.
[Equation]
The last line is because the non-decreasing distance between g ( x, θ g,T ) and g ( x (cid:48) , θ g,T ) ∀ x, x (cid:48) ∈ D leads to larger variance σ t when using a stationary and monotonic kernel.
Lemma 3.
The information gain for the points selected can be expressed in terms of the predictive variance.
If h T =
( h ( g ( x t , θ g,T ) , θ h,T ))
∈ R T :
[Equation]
Proof.
First, we get I ( y T ; h T ) =
H ( y T ) − 12 log | 2 πeσ 2 I | .
Then,
[Equation]
Since x 1 , ..., x T are deterministic conditioned on y T − 1 .
The result follows by induction.
Lemma 4.
The gap of the mutual information between collision-free g ( x t , θ g,T ) and unregularized g ( x t , θ g, 0 ) is
[Equation]
Here z t = g ( x t , θ g,T ) .
φ is the identiﬁcation of collided data points.
The result is a simple application of the chain rule of mutual information.
I [Equation] y T ; h T ) =
I ( y
T ; h ( z T , θ h,T )) corresponds to the information gain under fully regularized and assumed collision-free setting.
I ( y T , φ T ; h ( z T , θ h, 0 )) corresponds to information gain under unregularized setting.
Lemma 5.
Pick δ ∈ (0 , 1) and let β t be deﬁned as in Lemma 1.
Then, the following holds with probability ≥ 1 − δ , ∀ T ≥ 1 ,
[Equation]
Here C 1 :=
[Equation]
Proof.
We ﬁrst observe that
[Equation]
Combining the above inequality with Lemma 2, Lemma 3 and Lemma 4 completes the proof.
Now we are ready to prove Proposition 1.
Proof of Proposition 1. Proposition 1 is a simple consequence of Lemma 4 and Lemma 5 and Cauchy-Schwarz inequality.
A. P ROOF OF P ROPOSITION 2: R EGRET B OUND WITH L IPSCHITZ -C ONTINUOUS O BJECTIVE F UNCTION
We ﬁrst modify Lemma 5.7 and Lemma 5.8 in Srinivas et al.
(2010) since we are assuming the deterministic Lipschitz-continuity for h .
Use the same analysis tool Z t deﬁned as a set of discretization
Z t ⊂ Z where Z t will be used at time t in the analysis.
We choose a discretization Z t of size ( τ t ) d . so that ∀ z ∈ Z ,
[Equation]
where [ z ] t denotes the closest point in Z t to z .
Lemma 6.
Pick δ ∈ (0 , 1) and set β = 2 log( π t δ )
+
d log (cid:0) Lrdt 2 (cid:1) , where (cid:80) t ≥ 1 π − 1 t = 1 , π t > 0 .
Let τ t =
Lrdt 2 .
Hence then
[Equation]
holds with probability ≥ 1 − δ .
Proof.
Using the Lipschitz-continuity and equation 4, we have that
[Equation]
t By choosing τ t =
Lrdt 2 , we have | Z t | =
( Lrdt 2 ) d and ∀ z ∈ Z, | h [Equation] , θ h,t [Equation]
[Equation]
h [Equation] [Equation] Then using Lemma 1, we reach the expected result.
Based on Lemma 2 and Lemma 6, we could have the following result directly.
Lemma 7.
Pick δ ∈ (0 , 1) and set β = 2 log(2 π t δ )
+
d log (cid:0) Lrdt 2 (cid:1) , where (cid:80) t ≥ 1 π − 1 t = 1 , π t > 0 .
Then with probability ≥ 1 − δ , for all t ∈ N , the regret is bounded as follows:
[Equation]
Proof.
Using the union bound of δ/ 2 in both Lemma 2 and Lemma 6, we have that with probability 1 − δ :
[Equation]
which complete the proof.
Now we are ready to prove Proposition 2. Proof of Proposition 2.
Using Lemma 7, we have that with probability ≥ 1 − δ :
T
[Equation]
By Cauchy-Schwarz:
[Equation]
Finally, substitute π t with π 2 t 2 / 6 (since (cid:80) 1 /t 2 =
π 2 / 6 ).
Proposition 2 follows.
B D EMONSTRATION OF THE C OLLISION E FFECT B. V ISUALIZATION OF THE C OLLISION E FFECT IN THE L ATENT S PACE
We demonstrate the collision effect in the latent space. We train the same neural network on Feynman dataset with 101 data points which demonstrate the latent space after two retrains with the retrain interval set to be 50 data points. The regularized one employs DW LOC O , with the regularization parameter ρ = 1 e 5 , penalty parameter λ
= 1 e − 2 , retrain interval ˜ T , weighting parameter γ = 1 e − 2 and the base kernel set to be square exponential kernel.
The non-regularized one employs
LSO.
[Figure]
Besides, having fewer collisions in the latent space contribute to the optimization through improving the learned Gaussian process.
We observe in this comparison that the next point selected by the acquisition function of the regularized version is approaching the global optima, while the next point in the non-regularized version is to solve the uncertainty brought by the severe collision near the currently observed maxima.
B. T HE C OLLISION E FFECT ON P ROPER N EURAL N ETWORKS
In this section, we provide empirical results supporting the claim in section 3.3 that increasing the network complexity often does not help to reduce the collision in the latent space.
For Feynman task, we test the single-layer neural network, which consists of 10, 1001, 5000, or 6000 neurons with Leaky Relu activation functions.
For Max Area task, we test the three-layer dense neural network.
The ﬁrst layer consists of 50, 100, 1000 or 1500 neurons with Tanh activation functions.
The second layer consists of 50 neurons with Tanh activation functions.
The third layer consists of 10 neurons with Leaky Relu activation functions.
For Rastrigin-2D task, we test the single-layer neural network, which consists of 10, 100, 1000, or 5000 neurons with Leaky Relu activation functions.
For SPOKES task, we test the single-layer neural network, which consists of 10, 100, 1000, orneurons with Leaky Relu activation functions.
[Figure]
For the Water Converter task, we test the three-layer dense neural network.
The ﬁrst layer consists of 512, 1024, 2048, or 4096 neurons with Tanh activation functions.
The second layer consists of half of the ﬁrst layer’s neurons with Tanh activation functions.
The third layer consists of half of the second layer’s neurons with Leaky Relu activation functions.
We demonstrate the collision effect on regression task on Rastrigin-2D.
[Figure]
As illustrated in ﬁgure 7a, even after being sufﬁciently trained after 3000 iterations, there is still collisions in the latent space especially around the optima.
ﬁgure 7a shows that by regularizing the latent space, the ultimate MSE could also be improved.
C S UPPLEMENTAL M ATERIALS ON A LGORITHMIC D ETAILS
Our implementation of LOC O and DW LOC O is built upon the open source package GPytorch (Gardner et al., 2018).
The deep kernel is trained with back propogation.
We use the Adam (Kingma and Ba, 2014) optimizer with learning rate set to be 1 e − 2 .
Below we discuss the detailed conﬁguration of the underlying neural network and the choice of the key parameters used by the algorithm.
C. A LGORITHMIC D ETAILS ON N EURAL N ETWORK A RCHITECTURE
As the primary goal of our paper was to showcase the performance of a novel collision-free regularizer, we pick our network architectures to be basic multi-layer dense neural network.
We use a 4-layer dense neural network.
Its hidden layers consist of 1000, 500, 50 neurons respectively, each with Leaky Relu activation functions.
The output layer also uses Leaky Relu as its activation function and generates a 1-dimensional output.
[Figure]
Pre-training of the Neural Network We randomly select the points from the bottom of the dataset to avoid the pre-training simplifying the optimization task while helping with initializing the parameters of the model.
The points are used to initialize the neural network instead of being served as initial selections of the optimization task.
Without the pre-training stage, the latent embedding fed by the neural network to the gaussian process would be random.
The practical problem with such randomness could be a much larger variance for the results since it inﬂuences the following neural network training process and the optimization process. In practice, it’s always possible to collect pre-training datasets from relative domains when aiming at optimizing unknown objective functions. This pre-training method was also reported in the literature (Snoek et al., 2015). C. P ARAMETER CHOICES
We investigate the robustness of parameter choices of the regularization parameter λ on the Rastrigin 2D dataset.
We show the results in the ﬁgure below
D A DDITIONAL R ESULTS
Random EMbedding Bayesian Optimization (REMBO) (Wang et al., 2016) leverages simple random linear transformations to improve the efﬁciency in low-effective-dimension high-dimensional tasks.
We compare LOC O and DW LOC O with the performance of this random-embedding-based method and empirically exposed the failure case of REMBO when its modeling assumption does not hold (i.e. when dealing with dataset that has large effective dimensions).