Fast Bayesian Inference with Batch Bayesian Quadrature via Kernel Recombination
Anonymous Author(s)
AffiliationAddress email
Abstract
Calculation of Bayesian posteriors and model evidences typically requires numerical integration.
Bayesian quadrature (BQ), a surrogate-model-based approach tonumerical integration, is capable of superb sample efficiency, but its lack of parallelisation has hindered its practical applications.
In this work, we propose a parallelised(batch) BQ method, employing techniques from kernel quadrature, that possesses aprovably-exponential convergence rate.
Additionally, just as with Nested Sampling,our method permits simultaneous inference of both posteriors and model evidence.
Samples from our BQ surrogate model are re-selected to give a sparse set of samples, via a kernel recombination algorithm, requiring negligible additional time toincrease the batch size.
Empirically, we find that our approach significantly outperforms the sampling efficiency of both state-of-the-art BQ techniques and Nested
Sampling in various real-world datasets, including lithium-ion battery analytics.
Introduction
Many applications in science, engineering, and economics involve complex simulations to explain thestructure and dynamics of the process.
Such models are derived from knowledge of the mechanismsand principles underlying the data-generating process, and are critical for scientific hypothesisbuilding and testing.
However, dozens of plausible simulators describing the same phenomena oftenexist, owing to differing assumptions or levels of approximation.
Similar situations can be found inselection of simulator-based control models, selection of machine learning models on large-scaledatasets, and in many data assimilation applications [23].
In such settings, with multiple competing models, choosing the best model for the dataset is crucial.
Bayesian model evidence gives a clear criteria for such model selection.
However, computingmodel evidence requires integration over the likelihood, which is challenging, particularly whenthe likelihood is non-closed-form and/or expensive.
The ascertained model is often applied toproduce posteriors for prediction and parameter estimation afterwards.
There are many algorithmsspecialised for the calculation of model evidences or posteriors, although only a limited number of
Bayesian inference solvers estimate both model evidence and posteriors in one go.
As such, costlycomputations are often repeated (at least) twice.
Addressing this concern, nested sampling (NS)[59, 38] was developed to estimate both model evidence and posteriors simultaneously, and has beenbroadly applied, especially amongst astrophysicists for cosmological model selection [50].
However,
NS is based on a Monte Carlo (MC) sampler, and its slow convergence rate is a practical hindrance.
To aid NS, and other approaches, parallel computing is widely applied to improve the speed ofwall-clock computation.
Modern computer clusters and graphical processing units enable scientiststo query the likelihood in large batches.
However, parallelisation can, at best, linearly accelerate NS,doing little to counter NS’s inherently slow convergence rate as a MC sampler.
This paper investigates batch Bayesian quadrature (BQ) [52] for fast Bayesian inference.
BQ solvesthe integral as an inference problem, modelling the likelihood function with a probabilistic model(typically a Gaussian process (GP)).
Choosing an appropriate kernel and prior makes the integralsanalytical.
[30] proposed Warped sequential active Bayesian integration (WSABI), which adoptsactive learning to select samples upon uncertainty over the integrand.
WSABI showed that BQ withexpensive GP calculations could achieve faster convergence in wall time than cheap MC samplers.[65] introduced batch WSABI, achieving even faster calculation via parallel computing and becamethe fastest BQ model to date.
We improve upon these existing works for a large-scale batch case.
Background
Vanilla Bayesian quadrature The objective of BQ is to compute the following integrals and obtainsurrogate function of posterior:
[Equation]
where both ℓ ( x ) (e.g. a likelihood) and π ( x ) (e.g. a prior) are non-negative, and x ∈ R
d is a sample,and is sampled from prior x ∼ π ( x ) .
BQ solves the above integral as an inference problem, modellinga likelihood function by a GP.
The likelihood function ℓ ( x ) is modelled:
[Equation]
where X ∈ R n × d is the matrix of observed samples, y ∈ R n is the observed likelihood values, Kis the kernel.
Throughout the paper, we refer to a symmetric positive semi-definite kernel just as akernel.
Due to linearity, the mean and variance of the integrals are simply
[Equation]
In particular, the integral becomes analytic when π ( x ) is Gaussian and K is squared exponential kernel, K ( X , x )
= v (cid:112) | 2 π W |N ( X ; x, W ) , where v is kernel variance and W is the diagonal covariancematrix whose diagonal elements are the lengthscales of each dimension.
Since both the mean andvariance of the integrals can be calculated analytically, posterior and model evidence can be obtainedsimultaneously (see Supplementary).
Note that non-Gaussian prior is also possible to be selected viaimportance sampling.
Still, we use this combination throughout this paper for simplicity.
Warped Bayesian quadrature (WSABI) WSABI with linearisation approximation (WSABIL) sets the square-root GP for non-negativity with linearisation approximation of the transform˜ ℓ (cid:55)→ ℓ =
α + 12 ˜ ℓ 2 , where α = 0 .
8 × min ( y ) .
The square-root GP is defined as ˜ ℓ ∼ GP (˜ ℓ ; ˜ m y , ˜ C y ) ,and we have the following linear approximation:
[Equation]
Gaussianity implies the model evidence Z and posterior p ( x ) remain analytical (see Supplementary).
Kernel quadrature in general kernel quadrature (KQ) is the group of numerical integration rulesfor calculating the integral of function classes that form the reproducing kernel Hilbert space (RKHS).
With a KQ rule Q w , x given by weights w = ( w i ) ni =1 and points x = ( x i )
ni
=1 , we approximate theintegral by the weighted sum
[Equation]
where h is a function of RKHS H associated with the kernel K .
We define its worst-case error by
[Equation]
Q [Equation] , x ) [Equation]
=
[Equation]
x ) d x | .
Surprisingly, it is shown in [40] that we have
[Equation]
with X = x in the previous section.
Thus the point configuration in KQ with a small worst-case errorgives a good way to select points to reduce the integral variance in Bayesian quadrature.
Random Convex Hull Quadrature (RCHQ) Recall from (5) and (6) that we wish to approximatethe integral of a function h in the current RKHS.
First, we prepare n − 1 test functions φ 1 , . .
, φ n − 1based on M sample points using the Nyström approximation of the kernel: φ i ( x ) :=
u ⊤ i K
( x nys , x ) ,where u i ∈
R M is the i -th eigenvector of K ( x nys , x nys ) .
If we let λ i be the i -th eigenvalue of thesame matrix, the following gives a practical approximation [45]:
[Equation]
Next, we consider extracting a weighted set of n points ( w quad , x quad ) from a set of N points x rec withpositive weights w rec .
We do it by the so-called kernel recombination algorithm [46, 62], so thatthe measure induced by ( w quad , x quad ) exactly integrates the above test functions φ 1 , . . .
, φ n − 1 withrespect to the measure given by ( w rec , x rec ) [37].
In the actual implementation of multidimensional case, we execute the kernel recombination notby the algorithm [62] with the best known computational complexity
O ( C φ N + n 3 log( N/n ))
(where C φ is the cost of evaluating ( φ i ) n − 1 i =1 at a point), but the one of [46] using an LP solver(Gurobi [31] for this time) with empirically faster computational time.
We also adopt the randomized
SVD [33] for the Nyström approximation, so we have a computational time empirically faster than
O ( NM + M 2 log n + Mn 2 log( N/n ))
[37] in practice.
Related works
Bayesian inference for intractable likelihood Inference with intractable likelihoods is a longstanding problem, and a plethora of methods have been proposed.
Most infer posterior and evidenceseparately, and hence are not our fair competitors, as solving both is more challenging.
For posteriorinference, Markov Chain Monte Carlo [49, 36], particularly Hamilton Monte Carlo [39], is the goldstandard.
In a Likelihood-free inference context, kernel density estimation (KDE) with Bayesianoptimisation [32] and neural networks [29] surrogates are proposed for simulation-based inference[21].
In a Bayesian coreset context, scalable Bayesian inference [47], sparse variational inference[15, 16], active learning [55] have been proposed for large-scale dataset inference.
However, all of theabove only calculate posteriors, not model evidences.
For evidence inference, Annealed Importance
Sampling [51], and Bridge Sampling [6], Sequential Monte Carlo (SMC) [22] are popular, but onlyestimate evidence, not the posterior.
Bayesian quadrature The early works on BQ, which directly replaced the likelihood functionwith a GP [52, 53, 56], did not explicitly handle non-negative integrand constraints.
[54] introducedlogarithmic warped GP to handle the non-negativity of the integrand, and introduced active learningfor BQ, a method that selects samples based on where the variance of the integral will be minimised.[30] introduced square-root GP to make the integrand function closed-form and to speed up the computation.
Furthermore, they accelerated active learning by changing the objective from the varianceof the integral var[Z | y ] to simply the variance of the integrand var [ ℓ ( x ) π ( x )] .
[65] introduced first batch BQ.
[18] generalised warped GP for BQ, and proposed probit-transformation.
BQ has beenextended to machine learning tasks (model selection [17], manifold learning [25], kernel learning[34]) with new acquisition function (AF)s designed for each purpose.
This paper employs WSABI-Land its AF for simplicity, but our approach might be extended to incorporate the above advances.
Kernel quadrature There are a number of KQ algorithms from herding/optimisation [19, 4, 40]to random sampling [3, 5].
In the context of BQ, Frank-Wolfe Bayesian quadrature (FWBQ)
[9]using kernel herding has been proposed.
This method proposes to do BQ with the points given byherding, but the guaranteed exponential convergence O (exp( − cn )) is limited to finite-dimensional
RKHS, which is not the case in our setting.
For general kernels, the convergence rate drops to
O (1 / √ n ) [4].
Recently, a random sampling method with a good convergence rate was proposed forinfinite-dimensional RKHS [37].
Based on Carathéodory’s theorem for the convex hull, it efficientlycalculates a reduced measure for a larger empirical measure.
In this paper, we call it random convexhull quadrature (RCHQ) and use it in combination with the BQ methods.
Proposed Method: BASQ
4.1 General idea
We now introduce our algorithm, named Bayesian alternately subsampled quadrature (BASQ).
Kernel recombination for batch selection Batch WSABI [65] selects batch samples based on the
AF, taking samples having the maximum AF values greedily via gradient-based optimisation withmultiple starting points.
The computational cost of this sampling scheme increases exponentially withthe batch size and/or the problem dimension.
Moreover, this method is often only locally optimal [66].
We adopt a scalable, gradient-free optimisation via KQ algorithm based on kernel recombination[37].
Surprisingly, [40] pointed out the equivalence between BQ and KQ with optimised weights.
Kernel recombination can select n samples from the N candidate samples to efficiently reduce theworst-case error ( n ≪ N ).
The problem in batch BQ is selecting n samples from the probabilitymeasure f ( x ) that minimise integral variance.
When subsampling x quad ∼ f ( x ) , these are the same.[37] proposed an efficient algorithm based on kernel recombination, RCHQ, which automaticallyreturns a sparse set of n samples based on Carathéodory’s theorem.
The computational cost of batchsize n for our algorithm, BASQ, is lower than O ( NM + M 2 log n + Mn 2 log( N/n ))
Alternately subsampling The performance of RCHQ relies upon the quality of a predefined kernel.
Thus, we add BQ elements to KQ in return; making RCHQ an online algorithm.
Throughout thesequential batch update, we pass the BQ-updated kernels to RCHQ.
This enables RCHQ to exploitthe function shape information to select the best batch samples for the integral of the GP-modelledlikelihood function ℓ ( x ) .
This corresponds to the batch maximisation of the model evidence forℓ ( x ) .
Then, BQ optimises the hyperparameters based on samples from true likelihood ℓ true ( x ) , whichcorresponds to an optimised kernel preset for RCHQ in the next round.
These alternate updatescharacterise our algorithm, BASQ.
Importance sampling for uncertainty We added one more BQ element to RCHQ; uncertaintysampling .
RCHQ relies upon the quality of subsamples from the prior distribution.
However, sharp,multimodal, or high-dimensional likelihoods make it challenging to find prior subsamples that overlapover the likelihood distribution.
This typically happens in Bayesian inference when the likelihoodis expressed as the product of Gaussians, which gives rise to a very sharp likelihood peak.
Thelikelihood of big data tends to become multimodal [57].
Therefore, we adopt importance sampling ,gravitating subsamples toward the meaningful region, and correct the bias via its weights.
Wepropose a mixture of prior and an AF as a guiding proposal distribution .
The prior encourages global(rather than just local) optimisation, and the AF encourages sampling from uncertain areas for fasterconvergence.
However, sampling from AF is expensive.
We derive an efficient sparse Gaussianmixture sampler.
Moreover, introducing square-root warping [30] enables the sampling distributionto factorise, yielding faster sampling.
true likelihood prior posterior
GP-modelledlikelihood samplingfunction updatedGP subsamples x
~ g ( x ) (f) Recombination
Log MAE sparse -1.
true r
Log MAE (c) (b) (d)
L og
M A E o f i n t e g r a l
L og w a ll ti m e ( s )
L og
M A E o f i n t e g r a l
L og nu m b e r o f s a m p l e s
Log batch size
[Figure]
time
[Figure]
(e) Uncertainty Sampling
[Figure]
Summary of contribution We summarised the key differences between batch WSABI and BASQin Figure 1. 2 (a) shows that BASQ is more scalable in batch size.
(b) clarifies that BASQ can sample10 to 100 times as many samples in the same time budget as WSABI, supported by the efficientsampler and RCHQ.
(c) states the convergence rate of BASQ is faster than WSABI, regardless ofcomputation time.
(d) demonstrates the combined acceleration in wall time.
While the batch WSABIreached 10 − 1 after 1,000 seconds passed, BASQ was within seconds.
(e) and (f), visualised how
RCHQ selects sparse samples than batch WSABI.
This clearly explains that gradient-free kernelrecombination is better in finding the global optimal than multi-start optimisation.
These resultsdemonstrate that we were able to combine the merits of BQ and KQ.
We further tested with varioussynthetic datasets and real-world tasks in the fields of lithium-ion batteries and material science.
Moreover, we mathematically analyse the convergence rate with proof in Section 6.
4.2 Algorithm
Table 1 illustrates the pseudo-code for our algorithm.
Rows 4 - 10 correspond to RCHQ, and rows11 - 15 correspond to BQ.
We can use the variance of the integral var [ Z | y ] as a convergence criterion.
For hyperparameter optimisation, we adopt the type-II maximum likelihood estimation (MLE) tooptimise hyperparameters via L-BFGS [14] for speed.
Importance sampling for uncertainty We define the distribution of interest as f ( x ) = π ( x ) , prior,and the proposal distribution g ( x ) is a mixture of prior π ( x ) and acquisition function A ( x ) :
[Equation]
where r is the partition ratio of the proposal distribution, w IS is the weight of the importance sampling.
To ensure global optimisation, we mix the distributions f
and g with the partition ratio r . Larger rencourages uncertainty sampling to explore other peaks, whereas r = 0 is the vanilla MC sampler.
Notation : x init : initial guess, k : a convergence criterion, n : batch size, M : Nyström sample size, N : recombination sample size, ℓ : GP-modelled likelihood, ℓ true : true likelihood, π ( x ) : prior, p ( x ) : posterior, A ( x ) : AF, K : kernel,
S nys , S rec : samplers for Nyström and recombination, respectively
Input : prior π ( x ) , true likelihood function ℓ true
Output : posterior p ( x ) , the mean and variance of model evidence E
[ Z | y ] , var [ Z | y ] 1: C L y , A ( x ) ← InitialiseGPs ( x init )
# Initialise GPs with initial guess 2: S nys , S rec ← SetSampler ( A ( x ) , π ( x ))
# Set samplers 3: while var [ Z | y ] > k : 4: x nys ∼ S nys ( M )
# Samples M points for test function φ 5: ( w rec , x rec ) ∼ S rec ( N ) # Samples N points for recombination 6: φ 1 , ..., φ n − 1 ← Nyström ( x nys , C L y )
# Define n − 1 test functions 7: solve a kernel recombination problem 8:
Find w quad ≥ 0 and x quad ⊂ x rec 9: s.t. w ⊤ quad φ i ( x quad ) =
w ⊤ rec φ i ( x rec ) , w ⊤ quad 1 = w ⊤ rec10: return x quad #
The sparse set of n samples 11: y = Parallel ( ℓ true ( x quad ))
# Parallel computing of likelihood 12:
K ← Update ( x quad , y )
# Train GPs 13:
C L y , A ( x ) ← OptHypersThenUpdate ( K )
# Type II MLE optimisation 14:
S nys , S rec ← ResetSampler ( A ( x ) , π ( x ))
# Reset samplers with the updated A ( x ) 15:
Efficient sampler Sampling from A ( x ) , a mixture of Gaussians, is expensive, as some mixtureweights are negative, preventing the usual practice of weighted sampling from each Gaussian.
As theunwarped kernel C L y ( x, x ) is also computationally expensive, we adopt a factorisation trick :
[Equation]
where we have changed the distribution of interest to f ( x ) = |
˜ m y ( x ) | π ( x ) .
This is doubly beneficial.
Firstly, the distribution of interest f ( x ) will be updated over iterations.
The previous f ( x ) =
π ( x )means the subsample distribution eventually obeys prior, which is disadvantageous if the prior doesnot overlap the likelihood sufficiently.
On the contrary, the new f ( x ) narrows its region via | ˜ m y ( x ) | .
Secondly, the likelihood function changed to | ˜ ℓ ( x ) | , thus the kernels shared with RCHQ changed intocheap warped kernel ˜
C y ( x, x ) .
This reduces the computational cost of RCHQ, and the sampling costof A ( x ) .
Now A ( x ) =
˜
C y ( x ) π ( x ) , which is also a Gaussian mixture, but the number of componentsis significantly lower than the original AF (9).
As ˜ C y ( x ) is positive, the positive weights of the
Gaussian mixture should cover the negative components.
Interestingly, in many cases, the positiveweights vary exponentially, which means that limited number of components dominate the functionalshape.
Thus, we can ignore the trivial components for sampling.
Then we adopt SMC [43] tosample A ( x ) .
We have a highly-calibrated proposal distribution of sparse Gaussian mixture, leadingto efficient resampling from real A ( x ) (see Supplementary).
Variants of proposal distribution Although we can mathematically derive the efficacy of theabove proposal distribution g ( x ) , sometimes we have prior information not included in the aboveprocedure.
To incorporate human inductive bias, we propose two additional “biased” proposal distributions.
The first case is where we know both the maximum likelihood points and the likelihood’sunimodality.
This is typical in Bayesian inference because we can obtain (sub-)maximum pointsvia a maximum a posteriori probability (MAP) estimate.
In this case, we know exploring aroundthe perfect initial guess is optimal rather than unnecessarily exploring an uncertain region.
Thus,we introduce the initial guess believer (IGB) proposal distribution, g IGB ( x ) .
This is written asg IGB ( x ) = (1 − r )
π ( x ) + r (cid:80) i =
1 w i, IGB N ( x ; X i , W ) , where w i, IGB = { 0 if y i ≤ 0 , else 1 } , X i ∈
X .
This means exploring only the vicinity of the observed data X .
The second case is where weknow the likelihood is multimodal.
In this case, determining all peak positions is most beneficial.
Thus more explorative distribution is preferred.
As such, we introduce the uncertainty believer (UB)proposal distribution, g UB ( x ) .
This is written as g UB ( x ) =
A ( x ) , meaning pure uncertainty sampling.
To contrast the above two, we term the proposal distribution in Eq.
(8) as integral variance reduction(IVR)
g IVR ( x ) .
Experiments
Given our new model BASQ, with three variants of the proposal distribution, IVR, IGB, and UB, wenow test for speed against MC samplers and batch WSABI.
We compared with MultiNest NS [59, 24]and Dynamic NS [38], and Reactive NS with MLFriends [10, 11], coded with dynesty [60] and UltraNest [13] library.
According to the review [12], MLFriends is the state-of-the-art NS sampler todate.
The code is implemented based on BayesQuad [64], GPy [28], NumPy [35] SciPy [63], Gurobi[31], and code around kernel recombination [20, 37] with additional modification.
All experiments onsynthetic datasets were averaged over 10 repeats, computed in parallel with multicore CPUs, without GPU for fair comparison.
The posterior distribution of NS was estimated via KDE with weightedsamples [27].
For maximum speed performance, batch size was optimised for each method in eachdataset, in fairness to the competitors.
Batch WSABI needs to optimise batch size to balance thelikelihood query cost and sampling cost, because sampling cost increases rapidly with batch size, asshown in Figure 1(a).
Therefore, it has an optimal batch size for faster convergence.
By wall timecost, we exclude the cost of integrand evaluation; that is, the wall time cost is the overhead cost ofbatch evaluation.
Details can be found in the Supplementary.
5.1 Synthetic problems
We evaluate all methods on three synthetic problems.
The goal is to estimate the integral and posteriorof the likelihood modelled with the highly multimodal functions.
Prior was set to a two-dimensionalmultivariate normal distribution, with a 0 mean vector, and covariance whose diagonal elements are2.
The optimised batch sizes for each methods are BASQ: 100, batch WSABI: 16.
The syntheticlikelihood functions are cheap (0.5 ms on average).
This is advantageous setting for NS:
Within 10seconds, the batch WSABI, BASQ, and NS collected 32, 600, 23225 samples, respectively.
As forthe metrics, posterior estimation was tested with Kullback-Leibler (KL) upon random 10,000 samplesfrom true posterior.
Evidence was evaluated with MAE, and ground truth was derived analytically.
Likelihood functions Branin-Hoo [41] is 8 modal function in two-dimensional space.
Ackley [61]is a highly multimodal function with point symmetric periodical peaks in two-dimensional space.
Oscillatory function [26] is a highly multimodal function with reflection symmetric periodical peaksof highly-correlated ellipsoids in two-dimensional space.
5.2 Real-world dataset
We consider three real-world applications with different computation times. The first two aresimulator-based likelihoods, and the third is a hierarchical GP for high-resolution time-series analysis.
We cannot obtain ground-truth posterior and model evidence. Thus, we adopted the empirical metric.
For the posterior, we can calculate the true conditional posterior distribution along the line passingthrough ground truth parameter points.
Then, evaluate the posterior with root mean squared error(RMSE) against 50 test samples for each dimension.
For integral, we compare the model evidenceitself.
With regard to the sampling speed, querying these likelihoods is expensive. Therefore, thereis no significant difference between batch WSABI and BASQ in the sample size that can earn per
L og K L d i nv e r g e n ce
L og M
A E o f i n t e g r a l
Branin-Hoo Function
Ackley Function
Oscillatory Function wall time (s) wall time (s) wall time (s)
L og R M
S E o f po s t
e r i
o r
N e g a ti v e
L og e v i d e n ce wall time (s) wall time (s)
Battery simulator
Phase-field model
BASQ-IVR
BASQ-IGB
BASQ-UB batch WSABI
NS (MultiNest)
NS (dynamic)
NS (MLFriends)
Hierarchical GP wall time (s) wall time.
However, NS is based on rejection sampling.
Thus, more than 50% of queried samplesare dismissed on average, meaning the sample size of NS is roughly half of BASQ.
Each simulatorreduced its spatial and time resolution for simplicity.
Still, the time becomes much longer when itincreases for accuracy.
Similarly, the number of data for the hierarchical GP model was reduced toapproximately 40,000.
Yet, the original data size is roughly 1,000 times. The batch sizes are BASQ:32, batch WSABI: 8. For the details, see Supplementary. Parameter estimation of the lithium-ion battery simulator
The simulator is the SPMe [48], 5estimating 3 simulation parameters at a given time-series voltage-current signal (the diffusivity oflithium-ion on the anode and cathode, and the experimental noise variance).
Prior is modified to logmultivariate normal distribution from [2].
Each query takes 1.2 seconds on average.
Parameter estimation of the phase-field model
The simulator is the phase-field model [48], 254estimating 4 simulation parameters at given time-series two-dimensional morphological image(temperature, interaction parameter, Bohr magneton coefficient, and gradient energy coefficient).
Prior is a log multivariate normal distribution.
Each query takes 7.4 seconds on average.
Hyperparameter marginalisation of hierarchical GP model
The hierarchical GP model wasdesigned for analysing the large-scale battery time-series dataset from solar off-grid system field data[1].
For fast estimation of parameters in each GP, the recursive technique [58] is adopted.
The taskis to marginalise 5 GP hyperparameters at given hyperprior, which is modified to log multivariatenormal distribution from [1].
Each query takes 1.1 seconds on average.
5.3 Results
We find BASQ consistently delivers strong empirical performance, as shown in Figure 2.
Onall benchmark problems, BASQ-IVR, IGB, or UB outperform all baseline methods.
Interestingly,the superiority is regardless of the computational cost of the likelihood.
This suggests that BASQcould be a generally fast Bayesian solver as far as we investigated.
In the multimodal setting of thesynthetic dataset, explorative UB proposal distribution exceeds, whereas IVR and IGB work in asimulator-based likelihood setting.
This is mainly because the parameter posteriors are not highlymultimodal in the battery simulator case and the prior is well-tuned from [2].
When comparing eachproposal distribution, only BASQ-IVR consistently outperforms WSABI.
Our results support thegeneral use of BASQ-IVR, or BASQ-UB if the likelihood is known to be highly multimodal.
Convergence analysis for BASQ
Note that the kernel K on R d in this section refers to the given covariance kernel of GP at each step.
We discuss the convergence of BASQ in one iteration.
We consider the importance sampling: Let fbe a probability density on R d and g be another density such that f = λg with a nonnegative functionλ .
Let us call such a pair, ( f, g ) , a density pair with weight λ .
We approximate the kernel K with K
= (cid:80) n − 1 i =1 c
i φ i ( x ) φ i ( y ) .
In general, we can apply the kernel recombination algorithm [46, 62] with the weighted sample ( w rec , x rec ) to obtain a weightedpoint set ( w quad , x quad ) of size n satisfying w ⊤ quad φ i ( x quad ) =
w ⊤ rec φ i ( x rec ) ( i = 1 , . . .
, n − 1 )and w ⊤ quad 1 = w ⊤ rec 1 .
By modifying the kernel recombination algorithm, we can requirew ⊤ quad k 1 / 2 1 ( x quad ) ≤ w ⊤ rec k 1 / 2 1 ( x rec ) , where k 1 / 2 1 ( x ) :=
(cid:112) K ( x, x ) − K 0 ( x, x ) [37].
We call such( w quad , x quad ) a proper kernel recombination of ( w rec , x rec ) with K 0 .
8 We have the following guarantee (proved in Supplementary):
Theorem 1.
Suppose (cid:82) (cid:112) K
( x, x ) f ( x ) d x < ∞ , ℓ ∼ GP ( m, K ) , and we are given an ( n − 1) -dimensional kernel K 0 such that K 1 :
= K − K 0 is also a kernel.
Let ( f, g ) be a density pairwith weight λ .
Let x rec be an N -point independent sample from g and w rec :
= λ ( x rec ) .
Then, if( w quad , x quad ) is a proper kernel recombination of ( w rec , x rec ) for K 0 , it satisfies [Equation] var [ Z [Equation]
| x quad ]
[Equation]
where Z f := (cid:82) ℓ ( x ) f ( x ) d x and C K,f,g := (cid:82) K ( x, x ) λ ( x ) f ( x ) d x − (cid:82)(cid:82) K
( x, y ) f ( x ) f ( y ) d x d y .
The above approximation has one source of randomness which stems from sampling N points x recfrom g .
One can also apply this estimate with a random kernel and thereby introduce another sourceof randomness.
In particular, when we use the Nyström approximation for K 0 (that ensures K 1 is akernel [37]), then one can show that (cid:82) K 1 ( x, x ) f ( x ) d x can be bounded by
[Equation]
where σ n is the n -th eigenvalue of the integral operator L 2 ( f ) ∋ h (cid:55)→ (cid:82) K (
· , y ) h ( y ) f
( y )
d x ,
K max := sup x K ( x, x ) .
However, note that unlike Eq.
(12), this inequality only applies with highprobability due to the randomness of K 0 ; see Supplementary for details.
If, for example, K is a Gaussian kernel on R d and f is a Gaussian distribution, we have σ n =
O (exp( − cn 1 /d )) for some constant c >
(see Supplementary).
So in (12) we also achieve anexponential rate as well.
It also works well with a moderate M in practise.
Note that unlike theprevious analysis [42], we do not have to assume that the space is compact.
Discussion
We introduced a batch BQ approach, BASQ, capable of simultaneous calculation of both modelevidence and posteriors.
BASQ demonstrated faster convergence (in wall-clock time) on bothsynthetic and real-world datasets, when compared against existing BQ approaches and state-of-the-art
NS.
Further, BASQ was proven to converge exponentially-fast under natural assumptions.
Althoughit scales to the number of data seen in large-scale GP experiments, practical BASQ usage is limitedto fewer than 16 dimensions (similar to many GP-based algorithms).
This is because filling highdimensional space requires exponentially many samples.
Employing a more efficient sampler, kernellearning, or importing ideas from coresets research may be productive lines of research.
As forthe mathematical proof, we do not incorporate batch and hyperparameter updates, which should beaddressed in future work.
The generality of our theoretical guarantee with respect to kernel anddistribution should be useful for extending the analysis to the whole algorithm.
Checklist
The checklist follows the references.
Please read the checklist guidelines carefully for information on how to 472 answer these questions.
For each question, change the default [TODO] to [Yes] , [No] , or [N/A] .
You arestrongly encouraged to include a justification to your answer , either by referencing the appropriate section of 474 your paper or providing a brief inline description.
For example:
• Did you include the license to the code and datasets?
[Yes] See Section ?? .
• Did you include the license to the code and datasets?
[No] The code and the data are proprietary.•
Did you include the license to the code and datasets?
[N/A] Please do not modify the questions and only use the provided macros for your answers.
Note that the Checklist 479 section does not count towards the page limit.
In your paper, please delete this instructions block and only keepthe Checklist section heading above along with the questions/answers below.
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contribu- tions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] See section 7(c)
Did you discuss any potential negative societal impacts of your work?
[N/A](d)
Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes]2.
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes] see section 6 andSupplementary(b) Did you include complete proofs of all theoretical results?
[Yes] see section 6 and Supplementary3.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] in the supplemental material
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] see section 5 and Supplementary
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] see section
[Table]
(d) Did you include the total amount of compute and the type of resources used (e [Table] g., type of GPUs,internal cluster, or cloud provider)?
[Yes] see section
[Table]
If you are using existing assets (e
[Table]
g., code, data, models) or curating/releasing new assets...
[Table]
(a) If your work uses existing assets, did you cite the creators?
[Yes] see section [Table]
(b) Did you mention the license of the assets?
[Yes] see section
[Table]
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]
[Table]
(d) Did you discuss whether and how consent was obtained from people whose data you’re us-
[Table]
ing/curating?
[Yes] see section
[Table]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
informa-
[Table]
tion or offensive content?
[N/A]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table] [Table]