SnAKe: Bayesian Optimization via Pathwise Exploration
Abstract
Bayesian Optimization is a very effective tool for optimizing expensive black-box functions.
Inspired by applications developing and characterizing reaction chemistry using droplet microfluidic reactors, we consider a novel setting where the expense of evaluating the function can increase significantly when making large input changes between iterations.
We further assume we are working asynchronously, meaning we have to select new queries before evaluating previous experiments.
This paper investigates the problem and introduces ‘Sequential Bayesian Optimization via Adaptive Connecting Samples’ (SnAKe), which provides a solution by considering large batches of queries and preemptively building optimization paths that minimize input costs.
We investigate some convergence properties and empirically show that the algorithm is able to achieve regret similar to classical
Bayesian Optimization algorithms in both synchronous and asynchronous settings, while reducing input costs significantly.
We show the method is robust to the choice of its single hyper-parameter and provide a parameter-free alternative.
Introduction
We introduce a method for black-box optimization that keeps step-wise input variations as small as possible.
A black-box function is expensive to evaluate (with respect to time or resources) and we do not have access to gradients.
Classically, black-box optimization finds an optimum by sequentially querying the function.
We study a variation of this problem, with two important differences.
First, we introduce the idea that large changes in inputs, between iterations, cause the function to become more expensive to evaluate.
Second, we do not assume observations are available immediately: delay between querying the function and getting a result leads to asynchronous decision making.
As a motivating example consider a droplet microfluidic reactor [48] (see Figure 6 in Appendix).
In such a reactor, we can quickly pump in chemicals, expose them to certain conditions, and collect the results of our experiments as they exit.
However, large changes in temperature mean that the reactor is no longer in steady-state, and this makes the evaluations unreliable until the system stabilizes.
Smaller changes can maintain a “quasi-steady state”, as the system is easier to stabilize.
Further, we have to wait for droplets to exit the reactor before obtaining observations.
More broadly, consider optimization requiring spatially continuous exploration.
Black-box optimization has been investigated for optimal placement of pollution sensors [17], and machine learning methods for detecting the sources of pollution have been developed [30].
Mobile pollution readers, such as those used by AirQo [ https://www.airqo.net/about ], would face similar challenges if black-box optimizers were used to track the pollution sources.
Samaniego et al.
[40] show a more concrete example where autonomous vehicles use black-box optimizers to monitor and track water contamination sources in the Ypacarai lake in Paraguay.
Classical Bayesian Optimization (BO) [20, 41] provides effective solutions to black-box optimization; however, physics-based limitations are usually not taken into account [49].
Typically BO follows a myopic approach, in that BO chooses the next query based only on the current state of the surrogate model.
Classical BO reduces uncertainty in unexplored areas and then returns to promising areas with no regard for the distance between consecutive query points.
This means that BO will incur very high input costs .
However, having zero changes in input space is obviously not a good solution.
After all we want to explore the search space to find the optimum point.
We seek an algorithm that preserves the essence of Bayesian Optimization.
Consider a scenario where we know a large number of inputs we want to query.
In this case, we could simply order the queries to attain the smallest input cost.
A good solution would simply require selecting a large number of queries, creating an ordering, and then following the path defined by the ordering.
Once we start obtaining new information, we could update our beliefs and update our optimization path.
This paper proposes S eque n tial Bayesian Optimization via A daptive ( K )Connecting Sampl e s (SnAKe).
Just as the snake grows by carefully eating items in the classic arcade game, a SnAKe optimization path grows from carefully adding queries to the evaluation path.
Related Work
‘Process-constrained Batch Bayesian Optimization’ [50], is the closest analog to our setting of BO with input costs.
Vellanki et al.
[50] navigate the physical limitations in changing the input space by fixing the complicated inputs for every batch.
Rather than fixing the difficult-to-change inputs, we penalize large input variations in line with the costs (time and resources) of changing conditions.
SnAKe decides when to make expensive input changes.
Recently, Ramesh et al.
[38] consider a similar cost setting to ours, inspired by applications to wind energy systems.
Waldron et al.
[51] compare the use of transient variable ramps (small input changes in a pre-determined manner) against a full steady-state design of experiments approach for learning parameters of chemical kinetic models.
They show that the transient approaches give less precise estimates, but much faster.
SnAKe combines the best of both worlds, automatically designing experiments while keeping input changes small.
‘Cost-aware Bayesian Optimization’ [28, 31, 43] optimizes regret with respect to cost per iteration.
These costs, which are fixed throughout the BO process, arise when some regions of the input domain are more expensive to evaluate than others.
Similar ideas appear in Multi-fidelity BO [21, 36] where cheap approximations for the objective function are used.
Bogunovic et al.
[6] combines cost-aware and level-set estimation approaches to solve environmental monitoring problems.
Diverging from prior works, our setting does not assume that different regions incur different costs.
Instead, SnAKe addresses the setting where the difference between adjacent inputs defines costs.
The query order changes the cost, therefore we focus on optimization paths instead of optimization sets.
The baseline of Cost-aware BO, Expected Improvement per unit Cost (EIpu), defined as
EI( x ) / Cost( x ) , is not directly applicable to our setting because the cost of not changing inputs, i.e., choosing x t = x
t − 1 , is zero.
We can instead use:
[Equation]
Three main issues: (a) Eq.
(1) introduces new hyper-parameter γ with no obvious way to choose it, (b) Eq.
(1) effectively penalizes the acquisition function far away from the data, so near local optima we only penalize exploration and may over-exploit, (c) asynchronous extensions have a tendency to encourage batch diversity, so we introduce a new trade-off between penalizing locally and far away.
As we shall see, SnAKe shows robust results with just one hyper-parameter.
Look ahead Bayesian Optimization [12, 15, 26] considers possible future queries to make the current choice less myopic.
This would be a great way of selecting the inputs to query, as we could simply look-ahead at what any classical BO would choose and then order the queries accordingly.
However, one is only able to look ahead for a few iterations, due to the computational complexity of looking too far ahead into the future [5, 53].
For example, Lee et al.
[29] combines Markov Decision Processes [37], look-ahead and cost-aware ideas, but is limited by the short time horizons rollout can handle.
Reinforcement Learning [47] can be used with longer time-horizons, but it requires access to a dedicated training environment.
A computationally cheaper alternative that allows us to select many inputs to query is Batch Bayesian
Optimization (BBO)
BBO is the setting where we are able to parallelize function evaluations, and as such we want to select multiple queries simultaneously.
González et al.
[15] and Jiang et al.
[19] link BBO with look-ahead BO, using a Local Penalization method [14] and Expected
Improvement (q-EI) [13], respectively.
Unfortunately, they restrict themselves to smaller batch sizes ( q ≤ 15 ) due to computational expense.
Asynchronous Bayesian Optimization [1, 22] addresses the problem of choosing queries while waiting for delayed observations.
Methods 3.1 Problem Set-up
We consider finding the maximum, x ∗ = arg max x ∈X f ( x ) , of a black-box function, f , where X is a compact subset of R d .
We assume f is continuously differentiable and expensive to evaluate.
We seek the optimum point while keeping the number of evaluations small.
We evaluate the function sequentially, over a discrete and finite number of samples, t = 1 , ..., T .
For every query, x t , we obtain a noisy observation of the objective, y t
= f ( x t ) + η t , where η t ∼ N (0 , s 2 ) is Gaussian noise.
We assume there is delay, t delay , between choosing a query and getting an observation.
So our data-set at iteration t is given by D t
= { ( x i , y i ) : i = 1 , ..., t − t delay − 1 } .
If we set t delay = 0 , we revert to classical sequential Bayesian Optimization, otherwise we are in an asynchronous setting.
Finally, we assume there is a known cost to changing the inputs to our evaluation, C ( x t , x t +1 ) .
We use simple regret, SR t = f ( x ∗ ) − max i =1 ,...,t f ( x i ) as the performance metric.
We want to minimize regret, or equivalently, maximize f , for the smallest possible cumulative cost, (cid:80) T − 1 t =1 C ( x t , x t +1 ) .
We note that the problem definition is invariant to cost scaling relative to the objective, because an optimal trajectory will be defined solely by the shape of the function.
Current cost-aware methods are not scale invariant, and as we shall see, SnAKe first chooses which points to query and then finds the most cost-efficient way of selecting them, making the method invariant to scaling of the cost.
As is common in Bayesian Optimization, we will model the black-box function by putting a Gaussian
Process (GP) prior on f ∼ GP ( µ 0 , σ 20 ) .
Since we have Gaussian noise, the posterior, f | D t is also a
GP, whose mean function, µ t ( · ) , and covariance function, κ t ( · , · ) , can be calculated analytically [39].
3.2 General Approach
For our general approach, we will seek to create a large batch of queries that we want to evaluate, and then plan-ahead a whole optimization path.
This is useful for three reasons: (1) it allows us to order the queries in a way that reduces input cost.
(2) It allows us to deal with any delay in getting observations, because we can pre-select future queries.
(3) It remains computationally feasible to plan-ahead even for very large time horizons.
We can follow this ordering or path until new information is available, after which we will update our path.
While we will provide a specific way of creating, planning and updating the paths, we note that the general ideas can be extended to suit different settings.
For example, Appendix C shows how to alter
SnAKe to simultaneously optimize multiple, independent, black-box functions.
3.3 Creating a Batch Through Thompson Sampling
We need to produce batches that are representative of the current state of the surrogate model.
In addition, the method should allow for big batch sizes, since we want to produce batches as big as our budget (which is usually much larger than the batch size most methods consider).
For example, in a micro-reactor, we might be interested in batches that contain hundreds of points [48].
Kandasamy et al.
[22] offers a promising solution where every point in the batch is independent.
The method is based on Thompson Sampling, which uses the GP’s inherent randomness to create a batch.
Each batch point is chosen by drawing a realization of the GP, and optimizing it.
The queries will fill out the space, and they are more likely to be on promising, and unexplored areas.
It should work very well in our context given we expect our initial batch sizes to be very large, so the sample should be representative of the current state of our surrogate model.
3.4 Creating a Path via the Travelling Salesman Problem
After selecting a batch of queries, P t = { x ( i ) t } ˜
Ti =1 , we order them.
We do this by embedding a graph into the batch, where the edge weights are the cost for changing one input to another.
We then find the shortest path that visits every point, i.e., we solve the Travelling Salesman Problem (TSP) [4, 8].
Section 3.9 discusses the computational cost.
Mathematically, we define the graph G =
( V, E, W ) , with V =
{ i ∈ 1 , ..., ˜ T : x ( i ) t ∈ P t } ,
E = { ( i, j ) : i, j ∈ 1 , ..., ˜ T } , and W = { w ij =
C ( x ( i ) t , x ( j ) t ) :
( i, j ) ∈ E } , where ˜ T is the number of batch samples.
We solve the TSP in G to obtain our latest optimization path.
A simple example would be to try to minimize the total distance travelled in input space, by selecting the Euclidean norm as cost C ( x t , x t +1 ) =
|| x t − x t +
|| .
3.5 Naively Updating the Optimization Path
After updating the GP with new observations, we want to use this information to update our path.
We propose updating our strategy by creating a new batch of points.
At iteration t , the remaining budget has size T − t .
We first propose sampling T − t queries through
Thompson Sampling, and then solving the Travelling Salesman Problem.
However, we show this leads to the algorithm getting ‘stuck’ in local optima.
This is because every time we re-sample, we naturally include some exploitation in the batch, and this exploitation will always be the next point chosen by the TSP–we will never reach the exploration algorithm steps.
Figure 1a shows an example.
[Figure]
3.6 Escape Analysis
To try and solve the convergence problem introduced by naive resampling, we briefly analyze it.
This analysis assumes we receive noise-less observations.
However, all sampling can be done in the presence of noise by calculating the corresponding posterior.
We note that this section will provide intuition for the next step of the algorithm, however, we are not providing rigorous theoretical justification for the step or proving any type of regret bounds.
We focus on Thompson Sampling.
Definition 3.1.
(Thompson Sample)
We say x ( i ) t is Thompson Sampled, and we write x ( i ) t ∼ τ t , if:
[Equation]
i
[Equation]
x ∈X f ( i ) t ( x ) , where f ( i ) t ( · ) ∼ GP ( µ t , κ t | D t )
Note that the sampling distribution GP ( µ t , κ t | D t ) changes at each iteration.
One particular concern, diagrammed in Figure 1a, is the possibility of the method being stuck in a certain area.
Let B δ ( a ) be a Euclidean δ -ball centered at a , and assume that x t − 1 ∈ B δ ( a ) .
We further assume two more things: (a) we resample the batch at every iteration, and (b) the TSP solution, or any approximation, will always choose a point in B δ ( a ) to be the next step, if there are any.
These assumptions represent the worst-case scenario for trying to escape a local optima.
Definition 3.2.
(Non-escape Probability)
We define the non-escape probability, p t , at iteration t , as the probability of a Thompson Sample falling into B δ ( a ) .
That is, for x ( i ) t ∼ τ t :
[Equation]
Let P t =
( x (1) t , ..., x ( T − t ) t ) ∼ τ t i.i.d. .
Of particular interest to us, is the number of ‘non-escapes’ in the sample, N t = |P
t ∩ B δ ( a ) | .
We are guaranteed to escape if no sample falls in B δ ( a ) , so we say we have fully escaped if x ( i ) t / ∈ B δ ( a ) ∀ i .
Remark 3.3.
Note that N t ∼ Binomial ( T − t, p t ) as all the samples are mutually independent, therefore the probability of fully escaping is P ( N t = 0) =
(1 − p t ) T − t .
This means we can only expect to fully escape if p t is very small.
Therefore we are interested in the behavior of p t as we gain more information about f in B δ ( a ) .
We now consider the circumstances under which p t becomes very small:
Areas without stationary points: Consider the case when our objective f does not contain a stationary point in B δ ( a ) .
Then the maximum of f on the closure of the ball, ¯ B δ ( a ) , must lie on the boundary of the ball.
In particular, if we assume that our GP model has no error in B δ ( a ) , and assuming continuity of sample paths, then the non-escape probability must be zero, p t = 0 .
Intuitively, the area itself contains enough information to ensure, with complete certainty, that the global optimum does not lie in the area.
We hope that, as we collect information inside areas without stationary points, p t → 0 , and we will eventually leave them with small probability of returning.
Appendix D.2 shows an example of this happening very fast.
Areas with stationary points: Areas with stationary points pose a much bigger problem.
We will restrict our arguments to local maxima, since this is where we have observed the problem.
Assume that f has a local optimum in B δ ( a ) , higher than any other we have observed before.
In this case, any sample taken from a Gaussian Process with no error in B δ ( a ) will have a local maximum inside
B δ ( a ) , and therefore it is possible that this local maximum is the global solution.
As we increase the information inside the area, p t is not guaranteed go down to zero.
This makes intuitive sense; the only way of knowing if a local optimum is not a global optimum is by sampling away from it–therefore with limited information we will allocate a certain probability to the global optimum being inside B δ ( a ) .
We include a clear example where p t → p > 0 in Appendix D.1.
Sampling consistently in a promising area is not necessarily a bad thing, indeed we want to exploit near possible global optimum candidates.
However, the question then becomes, how long will it take us to leave a local optimum?
Recall the probability of fully escaping is (1 − p t ) T − t , and therefore it will be increasing as t increases, even if p t is (almost) constant.
Remark 3.4.
Assume that p t → p > 0 , and that we have a high escape probability after t e iterations, i.e., (1 − p ) T − t e is large, leaving us T − t e iterations to explore the rest of the space.
If we increase our budget from T to T ′ , we will not have a high probability of escape until (1 − p ) T ′ − t
=
(1 − p ) T − t e , i.e., t = T ′ − T + t e .
This leaves us with T ′ − ( T ′ − T + t e )
= T − t e iterations to explore the rest of the space.
Note that this is independent of T ′ , meaning that increasing our budget does not increase our budget after leaving B δ ( a ) !
Rather, it only means we will be stuck in B δ ( a ) for a longer time.
This is very concerning as the method will be very myopic; if it finds a local optimum, it is likely that it will spend a very large amount of the budget exploiting it.
3.7 Escaping with ϵ -Point Deletion
Intuitively, the convergence problem stems from the constant resampling, because in every iteration resampling is reintroducing exploitative points.
We seek a way of making sure that the samples used by the algorithm to create paths take into account the previously evaluated designs.
Penalizing the samples via distances to previous queries seems like a simple solution, e.g., similar to local penalization [14].
However, penalization introduces the need to carefully tune the strength of the penalization so that we can still exploit optima and remain cost-effective – the parameterization is nontrivial.
Around a local maximum, if we assume that p t → p > 0 , then we can interpret p as the probability that the global maximum lies on the ball B δ ( a ) , as such, we focus on a method that has the property of exploiting the promising area for Tp iterations (that is, the total budget weighted by the probability).
To achieve this, we propose creating a total of T samples, and then deleting batch points that are similar to previously-explored points (i.e., we remove excess exploitation introduced by resampling).
Algorithm 1, which we term ϵ -Point Deletion, still allows us to exploit local optima if we sample many points near them, however, it should eventually move on.
Two points to note: (a) Point Deletion uses the Euclidean norm, and it is independent of the cost function.
This is because we are trying to escape local minima of simple regret.
(b) After the deterministic removals, it is likely the batch size is still larger than our remaining budget, so we balance it by randomly removing points from the batch.
Algorithm input: New proposed batch P t (size T ), set of already queried points Q t (size t ), and deletion distance ϵ for x ∈ Q t do ˜
d ← min x ′ ∈P t || x − x ′ ||
if ˜ d < ϵ then # find the closest point to the query x in the new batch ˜ x ← arg min x ′ ∈P t || x − x ′ || else # else pick a random sample
˜ x ← Random ( P t ) end if # remove said point from the batch
P t ← P t \ { ˜ x } end for output: A batch P t (size T − t )
ϵ -Point Deletion allows us to escape local optima by directly increasing the probability of fully escaping without changing p t .
Let q t be the number of previously queried points inside the ball, and set ϵ ≥ 2 δ .
Indeed, it follows that we will escape if N t ≤ q t , (as q t samples are guaranteed to be deleted due to previously queried points) which is much better than requiring N t = 0 .
Due to over-sampling, the expected number of ‘non-escapes’ is given by E [ N t ] =
p t T .
Around a local maximum, if we assume that p t → p > 0 , and hence, p t T ≈ pT , then we can reasonably expect an escape when q t = pT , that is, we will exploit the local maximum for approximately pT iterations.
This time we leave T − q t =
T (1 − p ) extra iterations to explore the remaining space!
Increasing the budget will benefit both the exploitation and the exploration instead of only the former (in contrast with Remark 3.4).
Figure 1 gives an empirical example where we use Point Deletion, with ϵ = 0 . 1 , to escape a local optimum.
We observe the expected behavior from our brief analysis.
For Point Deletion, we calculate the escape prediction as pT ≈ 74 , using ˆ p ≈ 0 . 74 , which we estimated in Appendix D.1.
We can see that without Point Deletion, we remain stuck in the first local optimum.
3.8 SnAKe
Algorithm 2 which we dub ‘Sequential Bayesian Optimization via Adaptive Connecting Samples’ (SnAKe), combines the ideas of previous sections.
Figure 2 diagrams the most important steps of
SnAKe.
Section 4 develops an effective, parameter-free alternative to the choice of ϵ .
Note there is no requirement for data to be available immediately following querying.
If t delay > 0 , we can simply stick to the latest path.
It works without modification on the asynchronous setting.
This is vital since we were inspired by chemical experiment design which can exhibit asynchronicity.
We also note that the problem is invariant to cost scaling, as the batch creation and point deletion steps are independent of the cost, and the solution to the Travelling Salesman Problem is also scale-invariant.
Algorithm 2 SnAKe input : Optimization Budget, T .
Deletion constant, ϵ .
begin : Create initial batch, P 0 , uniformly.
Choose starting point x 0 .
Q 0 ← { x 0 } .
Create initial path, S 0 , by solving TSP on P 0 .
for t = 1 , 2 , 3 , ..., T do
Check if any running evaluations are finished if there are new observations then
Update surrogate model
Create batch of size T using Thompson Sampling, P t −
P t − 1 ← ϵ -Point Deletion
( P t − 1 , Q t − 1 ) ˜
S ← TSP( P t , source = x t − 1 ) \
x t − 1 }
S ← Q t − 1 ∪ ˜ S end
if
Choose next query point from schedule: x t ← S t
Q t ← Q t − 1 ∪ { x t }
Evaluate f ( x t ) end for
[Figure]
The underlying function is Branin 2D.
The feasible set is [0 , 1] 2 , so we do not see all the samples or the complete path with this zoomed
3.9 Computational Considerations
We use Wilson et al.
[52] to efficiently sample the GP and optimize the samples using Adam [23].
The optimization step may be expensive, but it is highly parallelizable if required.
We heuristically solve the Travelling Salesman Problem with Simulated Annealing [24] and use an adaptive grid to reduce the number of samples away from our current input.
The grid consists of the closest N l samples to our current input; the remaining samples are assigned to one of N g points in a global grid.
Coarsifying the placement of the far-away points effectively ignores detailed paths far away from our current input.
These far-away points are not important to the problem, as long as we resample frequently enough.
We introduce two hyper-parameters ( N l , N g ), but they should not impact the solutions assuming N l is reasonably larger than any possible delay.
With these modifications, we were able to comfortably run all experiments in a CPU (2.5 GHz Quad-Core Intel Core i7), where
SnAKe shared a wall-time similar to Local Penalization methods.
Appendix E.1 gives more details.
Experimental Results
For all experimental results we report the mean and the standard deviation over 25 experimental runs.
We give the full implementation details and results in Appendix E and F respectively.
Classical BO methods are implemented using BoTorch [3] and GPyTorch [11].
The code to replicate all results is available online at https://github.com/cog-imperial/SnAKe .
In all experiments, we examine SnAKe for ϵ = 0 , 0 . 1 , and 1 .
We further introduce a parameter-free alternative by adaptively selecting ϵ to be the smallest length scale from the GP’s kernel, and denote it ℓ -SnAKe.
SnAKe proves to be robust to non-zero choices of ϵ .
We conjecture this happens because of Thompson Sampling’s exploitative nature [34], so all excess exploitation is concentrated on a very small area.
For ϵ =
we observe very low cost, at the expense of some regret.
Across all experiments we can see how SnAKe can traverse the search space effectively, constantly achieving good regret at low cost.
While other cost-aware methods may also achieve good performance, they are very inconsistent, suggesting they require careful tuning.
SnAKe achieves good, robust results with a single hyper-parameter.
We do highlight that SnAKe performs best when there are enough iterations for the samples to adequately fill the search space: this is highlighted by poor performance in Perm10D and by ordinary performance in low-budget experiments (see Appendix F, where we include performance comparisons for different budget sizes, T ).
[Figure] [Figure]
[Figure]
4.1 Synthetic Functions
Sequential BO
This section examines the performance of SnAKe against Classical BO algorithms.
We compare Expected Improvement (EI) [33], Upper Confidence Bound (UCB) [45], and Probability of Improvement (PI)
We compare against Expected Improvement per Unit Cost [43] (EIpu) as introduced in Eq.
(1), with γ = 1 , and with Truncated Expected Improvement [40] (TrEI).
We also introduce a simple baseline, where we create a random Sobol sample, and then arrange an ordering by solving the TSP, and never update the path again.
We do this for three classical benchmark functions, and six total in the Appendix F.1.1.
We set the cost function to be the 2-norm distance between the inputs.
Figure 4 shows the results in the top row.
Asynchronous BO We explore the asynchronous setting, comparing Local Penalisation with UCB (UCBwLP) [1, 14], Thompson Sampling (TS) [22], and the same Random baseline from the sequential setting.
We also test on EIpu with Local Penalisation (EIpuLP), with γ = 1 .
The results are shown in last two rows of Figure 4.
4.2 Reaction Control on SnAr Benchmark
We test our method on a real-world, SnAr chemistry benchmark [18].
We control four variables; equivalents of pyrrolidine, concentration of 2,4 dinitrofluorobenenze at reactor inlet, reactor temperature, and residence time.
We assume changing temperature, concentration and residence time incur an input cost, owing to the response time required for the reactor to reach a new steady state.
We assume
[Figure]
the reactor as a first-order dynamic system, where the response to changes in input is given by:
[Equation]
where s denotes the time after experiment x t is finished, ( x t ) i denotes the i th variable of the t th experiment, (∆ x t ) i
=
( x t +1 )
i − ( x t ) i , and α i is the system time constant.
We define quasi-steady state using an absolute residual β i , i.e., | ( x s )
i − ( x t +1 )
i | ≤ β i .
For input changes smaller than β
i , we assume a linear cost, defined by a parameter γ i .
Combining with the response time from (3):
[Equation]
We assume that we can change variables simultaneously, so the total input cost is simply the longest response time within a given set of input changes.
That is, C ( x t , x t +1 )
= max i ∈
I
c
C
i ( x t , x t +1 ) , where I c is the index set of the control variables.
We implement the simulation using the Summit package [10].
More details can be found in Appendix E.7.6.
We set a delay of t delay = 25 , and optimize for T = 100 iterations.
The results of the experiment can be seen in Figure 3.
4.3 Finding Contamination Sources in Ypacarai Lake
As a second real-world example that exhibits costly input changes, we investigate an alternative to the case study introduced by Samaniego et al.
Autonomous boats are used to monitor water quality in Ypacarai Lake in Paraguay.
We consider three different objectives, illustrated in Figure 5, created using the Schekel benchmark function (as in [40]).
Each objective could correspond to a measure of water quality, such as pH, turbidity, CO2 levels, or many more.
We consider the problem
[Table]
of finding the largest source of contamination in the lake by maximizing these objectives.
The cost between queries relates to the boat physically moving from one place to another.
We assume that the boats can travel 10 units, which corresponds to approximately 100km (fuel limitations), and no observation time delay.
We run optimization runs on each objective and shows the results in Table 1.
Conclusion and Discussion
This paper introduces and proposes a solution to the problem of optimizing black-box functions under costs to changes in inputs.
We empirically show that the regret achieved by our method is comparable to those of classical Bayesian Optimization methods and we succeed at achieving considerably lower input costs, while being particularly well suited to asynchronous problems.
Examples of further work include extending the algorithm so multiple ‘SnAKes’ can run in parallel, or extending it to the classical multi-objective setting, e.g., using variations of Thompson Sampling [7].
This setting, with input costs penalizing experimental changes, makes a major step towards automating new reaction chemistry discovery, e.g., in line with the vision of Lazzari et al.
We substantially decrease experimental cost with respect to classical black-box optimization, e.g., as used by Fath et al.
[9] and McMullen and Jensen [32].
In the real-life SnAR benchmark, SnAKe spends 45-55% of the cost while making similarly strong predictions to classical BO.
The synthetic benchmarks and
Ypacarai Lake example offer similar advantages.
Acknowledgments & Disclosure of Funding
JPF is funded by EPSRC through the Modern Statistics and Statistical Machine Learning (StatML)
CDT (grant no.
EP/S023151/1) and by BASF SE, Ludwigshafen am Rhein.
SZ was supported by an Imperial College Hans Rausing PhD Scholarship.
The research was funded by Engineering &
Physical Sciences Research Council (EPSRC) Fellowships to RM and CT (grant no.
EP/P016871/ and EP/T001577/1).
CT also acknowledges support from an Imperial College Research Fellowship.
We would like to thank the reviewers and meta-reviewers throughout the whole peer review process for their time, attention, and ideas, which led to many improvements in the paper.
Discussions with colleagues in the Imperial Department of Computing led to the ideas of ϵ -Point Deletion and the nonparametric ℓ -SnAKe.
Linden Schrecker improved our understanding of the chemistry application.
A Motivating Example
Figure 6 shows a graphic of a droplet microfluidic reactor [48] which is our main motivating example.
[Figure]
N
Input Chemical Composition
Microreactor conditions, e.g. Droplet flow rate, Light exposure, Heat exposure
SYSTEM INPUTS
SYSTEM OUTPUTS
Online outputs, e.g. Fluorescence,
Confocal microscopy, Infrared camera
Offline outputs, e.g.
Chromatographic assays
[Figure]
B General Approach
While we present SnAKe as a detailed algorithm, the ideas behind it are far more general.
For example, in Section C we create our batch by taking samples from three different GPs.
Algorithmpresents a more general algorithm that could be used for any similar task.
Algorithm 3 General Ordering-Based Optimization input : Optimization budget of T samples.
Method for creating batch of queries.
Method for creating an ordering from a batch of queries.
Method for updating paths.
begin : Create initial batch of size T and ordering, S for t = 1 , 2 , 3 , ..., T do if there is new information then
Update surrogate model
Choose a batch of new points to query
Create a new path, ˜ S
S ← ˜ S end if
Choose next query point from ordering: x t ← S t
Evaluate f ( x t ) end for
C Ypacarai Lake Experiment
A second motivating example for our research comes from optimization requiring spatially continuous exploration.
In this section, we examine the problem more closely, and show that SnAKe can easily be extended to optimize multiple black-box functions simultaneously in the same search space.
The problem in question is an adaptation of the case study introduced by Samaniego et al.
The application is to use autonomous boats to find the largest source of contamination in Lake Ypacarai in
Paraguay.
Figure 7 shows a visualization of the Lake, and three objectives we want to optimize over.
Each objective corresponds to a different measure of water contamination, examples might include pH, turbidity, CO2 levels, and more.
In Section 4.3 we explored optimizing over these examples individually.
[Table]
In this section, we extend the problem to optimize over all of these simultaneously .
This is important because it is inefficient to run multiple objective runs when we are exploring the same search space.
At every iteration we choose one query x t , and obtain three outputs f 1 ( x t ) , f 2 ( x t ) , and f 3 ( x t ) one for each objective.
We model each objective using an independent GP, meaning we can create Thompson Samples from any objective .
Therefore we extend SnAKe by creating the batch using a mixture of samples from each objective.
For the experiment we use the ratio 1:1:1.
For TrEI and EIpu, we adapt to the multi-objective case by optimizing the first objective until we have travelled 3.3 units (approximately 33km), after which we change to the subsequent objective.
Table 2 shows the results.
D Empirical Analysis of Escape Probability D.1 Areas with stationary points
Figure 8 estimates the non-escape probability (see Definition 3.2) from the interval A = [0 . 1 , 0 . 2] .
The optimization objective is a bi-modal function.
Once we have 15 samples in the interval [0 . 1 , 0 . 2] , we estimate the escape probability to converge to p ≈ 0 .
D.2 Areas without stationary points
We now repeat the same experiment as in section D.1, this time we change the interval to A = [0 . 0 , 0 . 1] which does not contain any stationary points.
One can observe a clear difference in the behavior of p t as we include more information.
This time, p t → 0 very fast.
D.3 Resampling vs Point Deletion
Figure 10 empirically confirms the analysis of Section 3.7 showing the effectiveness of ϵ -Point
Deletion, and displaying the effect of increasing the budget from 100 to 250 iterations.
[Figure]
14 Number of queries in [0.1, 0.2] 0.
[Figure] [Figure]
[Figure]
E s t i m a t e o f p t
[Figure]
which are chosen randomly with a uniform distribution in A ).
We repeat the experiment 10 times.
The left plot shows the underlying function and the Gaussian Process for 15 training points.
The right plot shows the evolution of our estimate as we increase training points inside A =
1] (we plot the mean of each run ± the standard deviation).
We can see that p t quickly converges to (almost) zero.
We almost guaranteed to fully escape the area after 15 time-steps, even for very large budgets.
E Implementation Details
This section outlines the implementation details and hyper-parameter choices for all the methods compared in the paper.
The code used in the paper is available at the following link: https: //github.com/cog-imperial/SnAKe .
E.1 Computational Considerations (Extended)
Unfortunately, Algorithm 2 (SnAKe) is computationally expensive in two aspects.
First, for large budgets, we may struggle to train and sample the GPs.
For our experiments, training was not an issue and we were able to use full model GPs.
However, we could use Sparse GPs [42] if needed.
The
Wilson et al.
[52] approach allows us to create GP samples efficiently, and possibly optimize them using gradient methods.
The sampling can be done in linear time ( after the GP has been trained).
We use the Wilson et al.
[52] method to create our samples, and then optimize the samples using Adam [23].
[Figure]
r a t i
[Figure]
[Figure] [Figure]
a
[Figure]
i
[Figure]
I t
[Figure]
r a t i o n
[Figure]
The second bottleneck is solving the TSP, which is NP-hard, and we may need to solve it almost at every iteration (for small values of t delay ).
There are heuristic solutions that give approximate solutions quickly.
We use Simulated Annealing [24] which grows linearly with the budget size, T .
For small budgets, simulated annealing should find good solutions, but it could struggle as the budget size increases.
To solve this, note that we do not actually require a super-specific solution to the problem: we are only expecting to query the first few points on a path before replacing it by an entirely new path.
We build an adaptive grid (at each iteration) consisting of two separate parts.
A very coarse grid, ξ global , covers most of the search space, and a very fine grid, ξ local,t , consisting of the N l samples closest to x t .
This allows us to define the grid ξ t = ξ global ∪ ξ local,t .
The remaining T − N l samples will be assigned to the closest point in ξ t (using the Euclidean distance).
The adaptive grid means we expect to have multiple samples assigned to the same point, specifically in the coarse areas of our grid.
But this is not important, because we expect our immediate attention will be in the area around our current input where there should be little to no repetition.
This will allow the algorithm to focus on testing solutions which are relevant to our problem.
The adaptive grid introduces two hyper-parameters: the size of the global and local grids, respectively
N g and N l .
Using this method, we run the TSP heuristics on a graph with at most min( N g + N l , T ) nodes.
For the experiments, we create the global grid using a simple Sobol grid [44].
E.2 Gaussian Processes
For every GP, we use the RBF Kernel with an output-scale, θ 0 :
[Equation]
where Θ = diag ( ℓ 1 , ..., ℓ d ) and ℓ i denotes the length-scale of the i th variable.
For the prior mean, we used a constant function with trainable value, µ 0 .
We implemented them all using the package
GPyTorch [11].
E.3 Training the hyper-parameters of the Gaussian Processes
Our method is well suited for physical systems.
Hence, we assume that there is good prior knowledge of the hyper-parameters.
In particular, we found it reasonable that each hyper-parameter would be given a lower and upper bound.
Normally, we would simply have a large initialization sample.
However, we believe this goes against the nature of the problem because we want to explore the space slowly to avoid large input costs.
So any type of initialization would be costly.
We simulate this in the following way: we first randomly sample max( T/ 5 , 10 d ) points from the d -dimensional input space, and train a GP on these data-points.
The hyper-parameters of the GP are optimized by maximizing the marginal log-likelihood [39] over 500 epochs using Adam [23] with a learning rate of 0.01.
The resulting hyper-parameters will correspond to the ‘educated guess’.
We then set the following bounds:
a) For the length-scale, the lower bound is half the educated guess, and the upper bound double the ‘educated guess’.
b) For the output-scale, θ 0 , the lower bound is half the educated guess, and the upper bound double the ‘educated guess’.
c)
For the initial mean, µ 0 , the lower bound is the educated guess minus a third of the initial variance, the upper bound is the educated guess plus a third of the initial variance.
d)
The noise parameter we simply set to be greater than 10 − 5 .
We (partially) enforce the constraints by setting a SmoothedBoxPrior on each parameter, with a variance of 0.001.
Finally, under the constraints defined above, we re-estimate the hyper-parameters every time we obtain 25 new observations.
To make sure all models receive fair initializations, we set the same seed for each run and function pair.
E.4 SnAKe
We used Simulated Annealing [24] to solve the Travelling Salesman Problem.
We implemented it using the NetworkX package [16].
We initalized the cycle with the ‘greedy’ sub-algorithm and used all default options.
We generated the Thompson Samples using the method introduced in Wilson et al.
[52] which we implemented ourselves.
To optimize the samples we used Adam [23] and PyTorch [35] over 10 d epochs, with a learning rate of 0.01.
We used 10 d multi-starts for each sample.
To create the samples, we used ℓ = 1024 Fourier bases.
For ℓ -SnAKe, we define an adaptive deletion constant, ϵ t = min( ℓ 1 ,t , ..., ℓ d,t ) , where ℓ i,t denotes the length scale of the i th variable at time t (recall we are re-training the hyper-parameters every newobservations, so the length scales change with time).
For the adaptive grid, we use N l = 25 local samples, and a corse global Sobol grid [44] of N g =points.
E.5 Classical Bayesian Optimization
We used BoTorch [3] to implement all methods in this section.
We optimized the acquisition functions across 150 epochs using Adam [23] with a learning rate of 0.0001 using 7500 random multi-starts.
E.5.1 Expected Improvement
Expected Improvement [33] optimizes the acquisition function:
[Equation]
where y best is our best observation so far.
Expected Improvement per Unit Cost [43] optimizes the acquisition function:
[Equation]
We set γ = 1 in all experiments to address the fact that not moving incurs zero cost.
Note that as γ → ∞ , the method will behave closer to normal Expected Improvement, and as γ → 0 the method will tend to stay closer and closer to the current input.
Truncated Expected Improvement [40] first maximizes the normal Expected Improvement acquisition function.
It then travels a distance of at most ℓ towards the proposed point, where ℓ is the GPs length-scale.
We exclude this method from synchronous SnAr, as it is not suited for general cost functions.
E.5.2 Upper Confidence Bound
Upper Confidence Bound [45] optimizes the acquisition function:
[Equation]
We set β t = 0 .
2 d log(2 t ) following Kandasamy et al.
E.5.3 Probability of Improvement
Probability of Improvement [25] optimizes the acquisition function:
[Equation]
E.5.4 Truncated Expected Improvement
Truncated Expected Improvement was developed by [40] trying to solve the task of using an automated boat to monitor the water quality of Ypacari Lake in Paraguay.
The method seeks to take distance travelled into account when doing Bayesian Optimization.
It does by selecting a point using Expected
Improvement, setting a direct path from our current input to the new point, and then truncating the path one length-scale away.
That is:
[Equation]
E.6 Asynchronous Bayesian Optimization E.6.1 Local Penalization
We use the penalization method as is described in González et al.
For the Lipschitz constant, we estimate it by calculating the gradient of µ t (using auto-differentiation) in a Sobol grid [44] of 50 d points and selecting L to be the maximum gradient in the grid.
UCB with Local Penalization We set β t = 0 .
2 d log(2 t ) and M = y best .
EIpu with Local Penalization We set γ = 1 .
E.6.2 Thompson Sampling
We use the sample procedure as in E.4, except we only optimize a single sample at every iteration.
E.7 Description of Benchmark Functions
We chose the benchmark functions to observe the behavior of SnAKe in a variety of scenarios.
More details of all the benchmark functions can be found in Surjanovic and Bingham [46].
E.7.1 Branin2D
The two-dimensional Branin function is given by.
The function has three global maximums:
[Equation]
where we optimize over X
= [ − 5 , 10] ×
We set a = − 1 , = 5 .
1 / (4 π 2 ) , c = 5 /π , r = 6 , s = − 10 , and t = 1 / (8 π ) .
E.7.2 Ackley4D
The four-dimensional Ackley function has a lot of local optimums, with the optima in the center of the search space.
The function is given by:
[Equation]
where we slightly shift the search space and optimize over X =
[ − 1 . 8 , 2 . 2] 4 .
This is to avoid having the optimum exactly at a point in the Sobol grid and giving SnAKe an unfair advantage.
We set a = 20 , = 0 . 2 , and c = 2 π .
E.7.3 Michaelwicz2D
The two-dimensional Michalewicz function is characterized by multiple local maxima and a lot of flat regions.
The function is given by:
[Equation]
where we set m = 10 and we optimize on the region
X
=
[0 , π ] 2 .
E.
.4 Hartmann
We select this function to see how the algorithms behave in similar functions as dimension increases.
We do three versions of the Hartmann function, with dimensions d = 3 , 4 , and 6 .
The equation is given by:
[Equation]
A
[Equation]
−
[Equation]

 where α = (1 , 1 . 2 , 3 , 3 . 2) T .
For d = 3 we use:
[Equation]
  3689 11704699 43871091 8732 5743  for d = 4 and d = 6 we use:
[Equation]
[Equation] [Equation]
[Equation] [Equation]
[Equation]
  1312 1696 82832329 4135 8307 3736 10042348 1451 3522 2883 30474047 8828 8732 5743 

They are all evaluated on the unit cube [0 , 1] d .
E.7.5 Perm10D
We select the 10-dimensional version of the Perm benchmark to test the capabilities of the algorithms in a very high-dimensional setting (by BO standards).
The equation is given by:
[Equation]

[Equation]
where we set β = 10 .
We evaluate it on X =
[ − 10 , 10] d .
E.7.6 SnAr4D
We implement the simulation using Summit [10].
We control temperature between 40 anddegrees, concentration from 0.1 to 0.5 moles per liter, and residence time between 0.5 and 2 minutes.
We set the cost parameters for temperature α temp = 5 , β temp = 1 , γ temp = 1 , for concentration α conc = 2 , β conc = 0 .
01 , γ conc = 1 , and for residence time α residence = 3 , β residence = 0 .
05 and γ residence = 1 .
We further optimize over the equivalents of pyrrolidine between 1 and 5 units, but we assume changing it incurs no input costs.
Since the SnAr benchmark is a multi-objective problem, we optimize a weighted sum of the two objectives:
[Equation]
where we set ω 1 = 10 − 4 and ω 2 = 0 .
We optimize over X =
[40 , 120] × [0 . 1 , 0 . 5] × [0 . 5 , 2] × [1 , 5] .
E.8 Ypacarai Implementation Details
As per [40] we model the Lake using the Shekel function which is given as:
[Equation]
For the objective we use m = 2 , m = 3 , and m = 2 .
The other parameters are:
[Equation]
Where C ( i ) and β ( i ) represent the parameters of the i th objective.
All objectives are optimized across on a subset of X =
The subset is defined by creating grid of points mapping a high-resolution black and white image of Lake Ypacarai onto X .
For simplicity, we assume the cost of moving from one point to another is simply the distance between the points, even though in practice we might need to take a longer route to avoid land.
For Truncated Expected Improvement, we simply project the truncation into the closest point grid, to avoid sampling points outside of the Lake.
F Full Experiment Results F.1 Tables of Results F.1.1 Synchronous Experiments
This section includes the full tables of results of the synthetic synchronous experiments.
The results are shown in Table 3 and 4.
F.1.2 Asynchronous Experiments
In this section we include the full table results of all asynchronous experiments.
The results are shown in Table 5 and 6.
[Table] [Table]
F.1.3 SnAr Benchmark
We ran additional experiments on the SnAr benchmark.
For the first one (which includes the example looked at in the main paper) we tested on a budget of T = 100 iterations for different values of t delay .
The results are included in Tables 7 and 8.
We also carried out synchronous results on the benchmark, for different budgets.
The results are included in Tables 9 and 10.
F.2 Graphs for results of Section 4.
We include the full graphs of the sequential Bayesian Optimization experiments.
Each row represents a different budget.
The left column shows the evolution of regret against the cost used, the middle column shows the evolution of regret with iterations, and the right column shows the evolution of the 2-norm cost.
The results encompass Figures 11 to 16.
The caption in each figure tells us the benchmark function being evaluated.
Each experiment is the mean ± half the standard deviation of 25 different runs.
[Table] [Table]
Michaelwicz2D
[Table] [Table]
[Table] [Table]
[Table]
F.3 Graphs for results of Section 4.
We include the full graphs of the asynchronous Bayesian Optimization experiments.
Each row represents a different budget.
The left column shows the evolution of regret against the cost used, the middle column shows the evolution of regret with iterations, and the right columns show the evolution of the 2-norm cost.
The results encompass Figures 17 to 28.
The caption in each figure tells us the benchmark function being evaluated, and the time-delay for getting observations back.
Each experiment is the mean ± half the standard deviation of 10 different runs.
F.4 Graphs for results of SnAr Benchmark (section 4.2)
Figure 29 includes the whole set of results of the SnAr benchmark in the asynchronous setting.
Figure 30 includes the results for the synchronous setting.
Each experiment is the mean ± half the standard deviation of 10 different runs.
regret comparable with Bayesian Optimization methods.
EIpu achieves the worst non-random performance in every instance
[Table]
[Figure] [Figure]
R
[Figure]
g r e t )
[Figure]
the budget, SnAKe outperforms two BO methods in regret, and outperforms all methods in cost.
ϵ = 0 gives the smallest cost of all at the expense of some regret.
[Figure] [Figure]
R
[Figure]
g r
[Figure]
t )
[Figure] [Figure]
g r e t )
[Figure]
( R e g r e t )
[Figure]
the evolution of the 2-norm cost.
SnAKe has regret comparable with other methods for all budgets (UCB outperforms the rest for larger ones).
SnAKe achieves significantly less cost at all budgets, this may be due to SnAKe exploring the many local optimums carefully.
The first column shows that SnAKe achieves by far the best regret for low cost.
In this example, SnAKe and EIpu have similar performance.
[Figure] [Figure]
e
[Figure]
[Figure] [Figure]
R
[Figure]
g r
[Figure]
t )
[Figure] [Figure]
g r
[Figure]
t )
[Figure]
( R e g r e t )
[Figure]
did not update the model, we would achieve a much better performance (as it would be equivalent to Random).
We observe this behavior in the asynchronous case, where having a time-delay helps the method perform better (see asynchronous
Ackley, Figure 20).
EIpu and TrEI perform well in this example, we conjecture this is because they are doing far more localized searches while SnAKe is trying to cover all the space available (which is very difficult in higher dimensions).
[Figure]
[Figure] [Figure]
[Figure] [Figure]
R e g r e t )
[Figure] [Figure]
r e t )
[Figure] [Figure]
[Figure]
Similar results to the case with smaller delay, see Figure 23.
[Figure] [Figure]
Similar results to other Hartmann benchmarks, see Figure 23.
[Figure] [Figure]
Similar results to other Hartmann benchmarks, see Figure 23.
[Figure]
R
[Figure] [Figure]
l o g
[Figure]
R e g r e t )
[Figure]
l o g ( R e g r e t )
[Figure]
l o g ( R e g r e t )
[Figure] [Figure]