Fast Bayesian Inference with Batch Bayesian Quadrature via Kernel Recombination
Anonymous Author(s)
AffiliationAddress email
Abstract
Calculation of Bayesian posteriors and model evidences typically requires numerical integration.
Bayesian quadrature (BQ), a surrogate-model-based approach tonumerical integration, is capable of superb sample efficiency, but its lack of parallelisation has hindered its practical applications.
In this work, we propose a parallelised(batch) BQ method, employing techniques from kernel quadrature, that possessesan empirically exponential convergence rate.
Additionally, just as with Nested Sampling, our method permits simultaneous inference of both posteriors and model evidence.
Samples from our BQ surrogate model are re-selected to give a sparse set ofsamples, via a kernel recombination algorithm, requiring negligible additional timeto increase the batch size.
Empirically, we find that our approach significantly outperforms the sampling efficiency of both state-of-the-art BQ techniques and Nested
Sampling in various real-world datasets, including lithium-ion battery analytics.
Introduction
Many applications in science, engineering, and economics involve complex simulations to explain thestructure and dynamics of the process.
Such models are derived from knowledge of the mechanismsand principles underlying the data-generating process, and are critical for scientific hypothesisbuilding and testing.
However, dozens of plausible simulators describing the same phenomena oftenexist, owing to differing assumptions or levels of approximation.
Similar situations can be found inselection of simulator-based control models, selection of machine learning models on large-scaledatasets, and in many data assimilation applications [24].
In such settings, with multiple competing models, choosing the best model for the dataset is crucial.
Bayesian model evidence gives a clear criteria for such model selection.
However, computingmodel evidence requires integration over the likelihood, which is challenging, particularly whenthe likelihood is non-closed-form and/or expensive.
The ascertained model is often applied toproduce posteriors for prediction and parameter estimation afterwards.
There are many algorithmsspecialised for the calculation of model evidences or posteriors, although only a limited number of
Bayesian inference solvers estimate both model evidence and posteriors in one go.
As such, costlycomputations are often repeated (at least) twice.
Addressing this concern, nested sampling (NS)[66, 41] was developed to estimate both model evidence and posteriors simultaneously, and has beenbroadly applied, especially amongst astrophysicists for cosmological model selection [58].
However,
NS is based on a Monte Carlo (MC) sampler, and its slow convergence rate is a practical hindrance.
To aid NS, and other approaches, parallel computing is widely applied to improve the speed ofwall-clock computation.
Modern computer clusters and graphical processing units enable scientists to query the likelihood in large batches.
However, parallelisation can, at best, linearly accelerate NS,doing little to counter NS’s inherently slow convergence rate as a MC sampler
This paper investigates batch Bayesian quadrature (BQ) [60] for fast Bayesian inference.
BQ solvesthe integral as an inference problem, modelling the likelihood function with a probabilistic model(typically a Gaussian process (GP)).
[32] proposed Warped sequential active Bayesian integration(WSABI), which adopts active learning to select samples upon uncertainty over the integrand.
WSABIshowed that BQ with expensive GP calculations could achieve faster convergence in wall time thancheap MC samplers.
[73] introduced batch WSABI, achieving even faster calculation via parallelcomputing and became the fastest BQ model to date.
We improve upon these existing works for alarge-scale batch case.
Background
Vanilla Bayesian quadrature While BQ in general is the method for the integration, the functionalapproximation nature permits solving the following integral Z and obtaining the surrogate functionof posterior p ( x ) simultaneously in the Bayesian inference context:
[Equation]
where both ℓ ( x ) (e.g. a likelihood) and π ( x ) (e.g. a prior) are non-negative, and x ∈ R
d is a sample,and is sampled from prior x ∼ π ( x ) .
BQ solves the above integral as an inference problem, modellinga likelihood function by a GP.
The likelihood function ℓ ( x ) is modelled:
[Equation]
where X ∈ R n × d is the matrix of observed samples, y ∈ R n is the observed likelihood values, K isthe kernel.
1 Due to linearity, the mean and variance of the integrals are simply
[Equation]
In particular, (3) becomes analytic when π ( x ) is Gaussian and K is squared exponential kernel,
K ( X , x ) = v (cid:112) | 2 π W |N ( X ; x, W ) , where v is kernel variance and W is the diagonal covariancematrix whose diagonal elements are the lengthscales of each dimension.
Since both the mean andvariance of the integrals can be calculated analytically, posterior and model evidence can be obtainedsimultaneously.
Note that non-Gaussian prior is also possible to be selected via importance sampling(see Supplementary).
Still, we use this combination throughout this paper for simplicity.
Warped Bayesian quadrature (WSABI) WSABI with linearisation approximation (WSABIL) adopts the square-root warping GP for non-negativity with linearisation approximation of thetransform ˜ ℓ (cid:55)→ ℓ
= α + 12 ˜ ℓ 2 .
The square-root GP is defined as ˜ ℓ ∼ GP (˜ ℓ ; ˜ m y , ˜ C y ) , and we havethe following linear approximation:
[Equation]
Gaussianity implies the model evidence Z and posterior p ( x ) remain analytical (see Supplementary).
Kernel quadrature in general kernel quadrature (KQ) is the group of numerical integration rulesfor calculating the integral of function classes that form the reproducing kernel Hilbert space (RKHS).
With a KQ rule Q w , X given by weights w = ( w i ) ni =1 and points X
=
( x i ) ni
=1 , we approximate theintegral by the weighted sum
[Equation]
where h is a function of RKHS H associated with the kernel K .
We define its worst-case error by
[Equation]
x
[Equation]
d x | .
Surprisingly, it is shown in [43] that we have
[Equation]
Thus, the point configuration in KQ with a small worst-case error gives a good way to select points toreduce the integral variance in Bayesian quadrature.
Random Convex Hull Quadrature (RCHQ) Recall from (5) and (6) that we wish to approximatethe integral of a function h in the current RKHS.
First, we prepare n − 1 test functions φ 1 , . .
, φ n − 1based on M sample points using the Nyström approximation of the kernel: φ i ( x ) :=
u ⊤ i K
( X nys , x ) ,where u i ∈
R M is the i -th eigenvector of K ( X nys , X nys ) .
If we let λ i be the i -th eigenvalue of thesame matrix, the following gives a practical approximation [50]:
[Equation]
Next, we consider extracting a weighted set of n points ( w quad , X quad ) from a set of N points X recwith positive weights w rec .
We do it by the so-called kernel recombination algorithm [54, 70], so thatthe measure induced by ( w quad , X quad ) exactly integrates the above test functions φ 1 , . . .
, φ n − 1 withrespect to the measure given by ( w rec , X rec ) [39].
In the actual implementation of multidimensional case, we execute the kernel recombination notby the algorithm
[70] with the best known computational complexity O ( C φ N + n 3 log( N/n ))
(where C φ is the cost of evaluating ( φ i ) n − 1 i =1 at a point), but the one of [54] using an LP solver(Gurobi [33] for this time) with empirically faster computational time.
We also adopt the randomized
SVD [35] for the Nyström approximation, so we have a computational time empirically faster than
O ( NM + M 2 log n + Mn 2 log( N/n ))
[39] in practice.
Related works
Bayesian inference for intractable likelihood Inference with intractable likelihoods is a longstanding problem, and a plethora of methods have been proposed.
Most infer posterior and evidenceseparately, and hence are not our fair competitors, as solving both is more challenging.
For posteriorinference, Markov Chain Monte Carlo [57, 38], particularly Hamilton Monte Carlo [42], is the goldstandard.
In a Likelihood-free inference context, kernel density estimation (KDE) with Bayesianoptimisation [34] and neural networks [31] surrogates are proposed for simulation-based inference[21].
In a Bayesian coreset context, scalable Bayesian inference [55], sparse variational inference[15, 16], active learning [63] have been proposed for large-scale dataset inference.
However, all of theabove only calculate posteriors, not model evidences.
For evidence inference, Annealed Importance
Sampling [59], and Bridge Sampling [6], Sequential Monte Carlo (SMC) [23] are popular, but onlyestimate evidence, not the posterior.
Bayesian quadrature The early works on BQ, which directly replaced the likelihood functionwith a GP [60, 61, 64], did not explicitly handle non-negative integrand constraints.
[62] introducedlogarithmic warped GP to handle the non-negativity of the integrand, and introduced active learningfor BQ, a method that selects samples based on where the variance of the integral will be minimised.[32] introduced square-root GP to make the integrand function closed-form and to speed up the computation.
Furthermore, they accelerated active learning by changing the objective from the varianceof the integral var[Z | y ] to simply the variance of the integrand var [ ℓ ( x ) π ( x )] .
[73] introduced first batch BQ.
[18] generalised warped GP for BQ, and proposed probit-transformation.
BQ has beenextended to machine learning tasks (model selection [17], manifold learning [26], kernel learning[36]) with new acquisition function (AF)s designed for each purpose.
This paper employs WSABI-Land its AF for simplicity, but our approach might be extended to incorporate the above advances.
Kernel quadrature There are a number of KQ algorithms from herding/optimisation [19, 4, 43]to random sampling [3, 5].
In the context of BQ, Frank-Wolfe Bayesian quadrature (FWBQ)
[9]using kernel herding has been proposed.
This method proposes to do BQ with the points given byherding, but the guaranteed exponential convergence O (exp( − cn )) is limited to finite-dimensional
RKHS, which is not the case in our setting.
For general kernels, the convergence rate drops to
O (1 / √ n ) [4].
Recently, a random sampling method with a good convergence rate was proposed forinfinite-dimensional RKHS [39].
Based on Carathéodory’s theorem for the convex hull, it efficientlycalculates a reduced measure for a larger empirical measure.
In this paper, we call it random convexhull quadrature (RCHQ) and use it in combination with the BQ methods.
(See Supplementary)
Proposed Method: BASQ
4.1 General idea
We now introduce our algorithm, named Bayesian alternately subsampled quadrature (BASQ).
Kernel recombination for batch selection Batch WSABI [73] selects batch samples based on the
AF, taking samples having the maximum AF values greedily via gradient-based optimisation withmultiple starting points.
The computational cost of this sampling scheme increases exponentially withthe batch size and/or the problem dimension.
Moreover, this method is often only locally optimal [74].
We adopt a scalable, gradient-free optimisation via KQ algorithm based on kernel recombination[39].
Surprisingly, [43] pointed out the equivalence between BQ and KQ with optimised weights.
Kernel recombination can select n samples from the N candidate samples to efficiently reduce theworst-case error ( n ≪ N ).
The problem in batch BQ is selecting n samples from the probabilitymeasure f ( x ) that minimise integral variance.
When subsampling X quad ∼ f ( x ) , these are the same.[39] proposed an efficient algorithm based on kernel recombination, RCHQ, which automaticallyreturns a sparse set of n samples based on Carathéodory’s theorem.
The computational cost of batchsize n for our algorithm, BASQ, is lower than O ( NM + M 2 log n + Mn 2 log( N/n ))
Alternately subsampling The performance of RCHQ relies upon the quality of a predefined kernel.
Thus, we add BQ elements to KQ in return; making RCHQ an online algorithm.
Throughout thesequential batch update, we pass the BQ-updated kernels to RCHQ.
This enables RCHQ to exploitthe function shape information to select the best batch samples for the integral of the GP-modelledlikelihood function ℓ ( x ) .
This corresponds to the batch maximisation of the model evidence forℓ ( x ) .
Then, BQ optimises the hyperparameters based on samples from true likelihood ℓ true ( x ) , whichcorresponds to an optimised kernel preset for RCHQ in the next round.
These alternate updatescharacterise our algorithm, BASQ.
(See Supplementary)
Importance sampling for uncertainty We added one more BQ element to RCHQ; uncertaintysampling .
RCHQ relies upon the quality of subsamples from the prior distribution.
However, sharp,multimodal, or high-dimensional likelihoods make it challenging to find prior subsamples that overlapover the likelihood distribution.
This typically happens in Bayesian inference when the likelihoodis expressed as the product of Gaussians, which gives rise to a very sharp likelihood peak.
Thelikelihood of big data tends to become multimodal [65].
Therefore, we adopt importance sampling ,gravitating subsamples toward the meaningful region, and correct the bias via its weights.
Wepropose a mixture of prior and an AF as a guiding proposal distribution .
The prior encourages global(rather than just local) optimisation, and the AF encourages sampling from uncertain areas for fasterconvergence.
However, sampling from AF is expensive.
We derive an efficient sparse Gaussianmixture sampler.
Moreover, introducing square-root warping [32] enables the sampling distributionto factorise, yielding faster sampling.
true likelihood prior posterior
GP-modelledlikelihood samplingfunction updatedGP subsamples x
~ g ( x ) (f) Recombination
Log MAE sparse -1.
true r
Log MAE (c) (b) (d)
L og
M A E o f i n t e g r a l
L og w a ll ti m e ( s )
L og
M A E o f i n t e g r a l
L og nu m b e r o f s a m p l e s
Log batch size
[Figure]
time (s)
[Figure]
(e) Uncertainty Sampling
[Figure]
Summary of contribution We summarised the key differences between batch WSABI and BASQin Figure 1. 4 (a) shows that BASQ is more scalable in batch size.
(b) clarifies that BASQ can sample10 to 100 times as many samples in the same time budget as WSABI, supported by the efficientsampler and RCHQ.
(c) states the convergence rate of BASQ is faster than WSABI, regardless ofcomputation time.
(d) demonstrates the combined acceleration in wall time.
While the batch WSABIreached 10 − 1 after 1,000 seconds passed, BASQ was within seconds.
(e) and (f), visualised how
RCHQ selects sparser samples than batch WSABI.
This clearly explains that gradient-free kernelrecombination is better in finding the global optimal than multi-start optimisation.
These resultsdemonstrate that we were able to combine the merits of BQ and KQ (see Supplementary).
We furthertested with various synthetic datasets and real-world tasks in the fields of lithium-ion batteries andmaterial science.
Moreover, we mathematically analyse the convergence rate with proof in Section 6.
4.2 Algorithm
Table 1 illustrates the pseudo-code for our algorithm.
Rows 4 - 10 correspond to RCHQ, and rows11 - 15 correspond to BQ.
We can use the variance of the integral var [ Z | y ] as a convergence criterion.
For hyperparameter optimisation, we adopt the type-II maximum likelihood estimation (MLE) tooptimise hyperparameters via L-BFGS [14] for speed.
Importance sampling for uncertainty Lemma 1 in the supplementary proves the optimal upperbound of the proposal distribution g ( x ) ≈ (cid:112) K
( x, x ) f ( x ) = (cid:113) C L y ( x, x ) f ( x ) , where f ( x ) :
=
π ( x ) .
However, sampling from square-root variance is intractable, so we linearised to g ( x ) ≈0 .
5(1 + C L y ( x, x ))
f ( x ) .
To correct the linearisation error, the coefficient 0.5 was changed into thehyperparameter r , which is defined as follows:
[Equation]
Notation : x init : initial guess, k : a convergence criterion, n : batch size, M : Nyström sample size, N : recombination sample size, ℓ : GP-modelled likelihood, ℓ true : true likelihood, π ( x ) : prior, p ( x ) : posterior, A ( x ) : AF, K : kernel,
S nys , S rec : samplers for Nyström and recombination, respectively
Input : prior π ( x ) , true likelihood function ℓ true
Output : posterior p ( x ) , the mean and variance of model evidence E
[ Z | y ] , var [ Z | y ] 1: C L y , A ( x ) ← InitialiseGPs ( x init )
# Initialise GPs with initial guess 2: S nys , S rec ← SetSampler ( A ( x ) , π ( x ))
# Set samplers 3: while var [ Z | y ] > k : 4:
X nys ∼ S nys ( M ) # Samples M points for test function φ 5: ( w rec , X rec ) ∼ S rec ( N ) # Samples N points for recombination 6: φ 1 , ..., φ n − 1 ← Nyström ( X nys , C L y )
# Define n − 1 test functions 7: solve a kernel recombination problem 8:
Find an n -point subset X quad ⊂
X rec and w quad ≥9: s.t.
w ⊤ quad φ i ( X quad ) =
w ⊤ rec φ i ( X rec ) , w ⊤ quad 1 = w ⊤ rec10: return X quad
# The sparse set of n samples 11:
y = Parallel ( ℓ true ( X quad ))
# Parallel computing of likelihood 12:
K ← Update ( X quad , y )
# Train GPs 13:
C L y , A ( x ) ← OptHypersThenUpdate ( K )
# Type II MLE optimisation 14:
S nys , S rec ← ResetSampler ( A ( x ) , π ( x ))
# Reset samplers with the updated A ( x ) 15:
E y y
←
Eqs
(4) 16: return p
, E
, var
Z | y ] where w IS is the weight of the importance sampling.
While r = 1 becomes the pure uncertaintysampling, r = 0 is the vanilla MC sampler.
Efficient sampler Sampling from A ( x ) , a mixture of Gaussians, is expensive, as some mixtureweights are negative, preventing the usual practice of weighted sampling from each Gaussian.
As thewarped kernel C L y ( x, x ) is also computationally expensive, we adopt a factorisation trick :
[Equation]
where we have changed the distribution of interest to f ( x ) = |
˜ m y ( x ) | π ( x ) .
This is doubly beneficial.
Firstly, the distribution of interest f ( x ) will be updated over iterations.
The previous f ( x ) =
π ( x )means the subsample distribution eventually obeys prior, which is disadvantageous if the prior doesnot overlap the likelihood sufficiently.
On the contrary, the new f ( x ) narrows its region via | ˜ m y ( x ) | .
Secondly, the likelihood function changed to | ˜ ℓ ( x ) | , thus the kernels shared with RCHQ changed intocheap warped kernel ˜
C y ( x, x ) .
This reduces the computational cost of RCHQ, and the sampling costof A ( x ) .
Now A ( x ) =
˜
C y ( x ) π ( x ) , which is also a Gaussian mixture, but the number of componentsis significantly lower than the original AF (9).
As ˜ C y ( x ) is positive, the positive weights of the
Gaussian mixture should cover the negative components.
Interestingly, in many cases, the positiveweights vary exponentially, which means that limited number of components dominate the functionalshape.
Thus, we can ignore the trivial components for sampling.
Then we adopt SMC [48] tosample A ( x ) .
We have a highly-calibrated proposal distribution of sparse Gaussian mixture, leadingto efficient resampling from real A ( x ) (see Supplementary).
Variants of proposal distribution Although (8) has mathematical motivation, sometimes we wishto incorporate prior information not included in the above procedure.
We propose two additional“biased” proposal distributions.
The first case is where we know both the maximum likelihoodpoints and the likelihood’s unimodality.
This is typical in Bayesian inference because we canobtain (sub-)maximum points via a maximum a posteriori probability (MAP) estimate.
In this case,we know exploring around the perfect initial guess is optimal rather than unnecessarily exploringan uncertain region.
Thus, we introduce the initial guess believer (IGB) proposal distribution,g IGB ( x ) .
This is written as g IGB ( x ) =
(1 − r ) π ( x ) + r (cid:80) i
=
1 w i, IGB N ( x ; X i , W ) , where w i, IGB ={ 0 if y i ≤ 0 , else 1 } , X i ∈
X .
This means exploring only the vicinity of the observed data X .
The second case is where we know the likelihood is multimodal.
In this case, determining all peakpositions is most beneficial.
Thus more explorative distribution is preferred.
As such, we introducethe uncertainty believer (UB) proposal distribution, g UB ( x ) .
This is written as g UB ( x ) =
A ( x ) ,meaning pure uncertainty sampling.
To contrast the above two, we term the proposal distribution in
Eq.
(8) as integral variance reduction (IVR)
g IVR ( x ) .
Experiments
Given our new model BASQ, with three variants of the proposal distribution, IVR, IGB, and UB, wenow test for speed against MC samplers and batch WSABI.
We compared with three NS methods[66, 25, 41, 10, 11], coded with [67, 13].
According to the review [12], MLFriends is the state-ofthe-art NS sampler to date.
The code is implemented based on [72, 29, 37, 71, 33], and code aroundkernel recombination [20, 39] with additional modification.
All experiments on synthetic datasetswere averaged over 10 repeats, computed in parallel with multicore CPUs, without GPU for faircomparison.
The posterior distribution of NS was estimated via KDE with weighted samples [28].
For maximum speed performance, batch size was optimised for each method in each dataset, infairness to the competitors.
Batch WSABI needs to optimise batch size to balance the likelihoodquery cost and sampling cost, because sampling cost increases rapidly with batch size, as shownin Figure 1(a).
Therefore, it has an optimal batch size for faster convergence.
By wall time cost,we exclude the cost of integrand evaluation; that is, the wall time cost is the overhead cost of batchevaluation.
Details can be found in the Supplementary.
5.1 Synthetic problems
We evaluate all methods on three synthetic problems.
The goal is to estimate the integral and posteriorof the likelihood modelled with the highly multimodal functions.
Prior was set to a two-dimensionalmultivariate normal distribution, with a 0 mean vector, and covariance whose diagonal elements are2.
The optimised batch sizes for each methods are BASQ: 100, batch WSABI: 16.
The syntheticlikelihood functions are cheap (0.5 ms on average).
This is advantageous setting for NS:
Within 10seconds, the batch WSABI, BASQ, and NS collected 32, 600, 23225 samples, respectively.
As forthe metrics, posterior estimation was tested with Kullback-Leibler (KL) upon random 10,000 samplesfrom true posterior.
Evidence was evaluated with MAE, and ground truth was derived analytically.
Likelihood functions Branin-Hoo [44] is 8 modal function in two-dimensional space.
Ackley [68]is a highly multimodal function with point symmetric periodical peaks in two-dimensional space.
Oscillatory function [27] is a highly multimodal function with reflection symmetric periodical peaksof highly-correlated ellipsoids in two-dimensional space.
5.2 Real-world dataset
We consider three real-world applications with expensive likelihoods, which are simulator-basedand hierarchical GP.
We adopted the empirical metric due to no ground truth.
For the posterior, wecan calculate the true conditional posterior distribution along the line passing through ground truthparameter points.
Then, evaluate the posterior with root mean squared error (RMSE) against 50test samples for each dimension.
For integral, we compare the model evidence itself.
Expensivelikelihoods makes the sample size per wall time amongst the methods no significant difference,whereas rejection sampling based NS dismiss more than 50% of queried samples.
The batch sizes are
L og K L d i nv e r g e n ce
L og M
A E o f i n t e g r a l
Branin-Hoo Function
Ackley Function
Oscillatory Function wall time (s) wall time (s) wall time (s)
L og R M
S E o f po s t
e r i
o r
N e g a ti v e
L og e v i d e n ce wall time (s) wall time (s)
Battery simulator
Phase-field model
BASQ-IVR
BASQ-IGB
BASQ-UB batch WSABI
NS (MultiNest)
NS (dynamic)
NS (MLFriends)
Hierarchical GP wall time (s) 90 1.
Parameter estimation of the lithium-ion battery simulator :
The simulator is the SPMe [56], 7estimating 3 simulation parameters at a given time-series voltage-current signal (the diffusivity oflithium-ion on the anode and cathode, and the experimental noise variance).
Prior is modified tolog multivariate normal distribution from [2].
Each query takes 1.2 seconds on average.
Parameterestimation of the phase-field model :
The simulator is the phase-field model [56], 8 estimating4 simulation parameters at given time-series two-dimensional morphological image (temperature,interaction parameter, Bohr magneton coefficient, and gradient energy coefficient).
Prior is a logmultivariate normal distribution.
Each query takes 7.4 seconds on average.
Hyperparametermarginalisation of hierarchical GP model
The hierarchical GP model was designed for analysingthe large-scale battery time-series dataset from solar off-grid system field data [1].
For fast estimationof parameters in each GP, the recursive technique [69] is adopted.
The task is to marginalise 5 GPhyperparameters at given hyperprior, which is modified to log multivariate normal distribution from[1].
Each query takes 1.1 seconds on average.
5.3 Results
We find BASQ consistently delivers strong empirical performance, as shown in Figure 2.
On allbenchmark problems, BASQ-IVR, IGB, or UB outperform baseline methods except in the batterysimulator evidence estimation.
The very low-dimensional and sharp unimodal nature of this likelihoodcould be advantageous for biased greedy batch WSABI, as IGB superiority supports this viewpoint.
This suggests that BASQ could be a generally fast Bayesian solver as far as we investigated.
In themultimodal setting of the synthetic dataset, BASQ-UB outperforms, whereas IVR does in a simulatorbased likelihoods.
When comparing each proposal distribution, BASQ-IVR was the performant.
Ourresults support the general use of IVR, or UB if the likelihood is known to be highly multimodal.
Convergence analysis
We analysed the convergence over single iteration on a simplified version of BASQ, which assumesthe BQ is modelled with vanilla BQ, without batch and hyperparameter updates.
Note that the kernel K on R d in this section refers to the given covariance kernel of GP at each step.
We discuss theconvergence of BASQ in one iteration.
We consider the importance sampling: Let f be a probabilitydensity on R d and g be another density such that f = λg with a nonnegative function λ .
Let us callsuch a pair, ( f, g ) , a density pair with weight λ .
We approximate the kernel K with K
= (cid:80) n − 1 i =1 c
i φ i ( x ) φ i ( y ) .
In general, we can apply the kernel recombination algorithm [54, 70] with the weighted sample ( w rec , X rec ) to obtain a weighted MATLAB [ point set ( w quad , X quad ) of size n satisfying w ⊤ quad φ i ( X quad ) =
w ⊤ rec φ i ( X rec ) ( i = 1 , . . .
, n − 1 )and w ⊤ quad 1 = w ⊤ rec 1 .
By modifying the kernel recombination algorithm, we can requirew ⊤ quad k 1 / 2 1 ( X quad ) ≤ w ⊤ rec k 1 / 2 1 ( X rec ) , where k 1 / 2 1 ( x ) :=
(cid:112) K ( x, x ) − K 0 ( x, x ) [39].
We call such( w quad , X quad ) a proper kernel recombination of ( w rec , X rec ) with K 0 .
9 We have the following guarantee (proved in Supplementary):
Theorem 1.
Suppose (cid:82) (cid:112) K
( x, x ) f ( x ) d x < ∞ , ℓ ∼ GP ( m, K ) , and we are given an ( n − 1) -dimensional kernel K 0 such that K 1 :
= K − K 0 is also a kernel.
Let ( f, g ) be a density pairwith weight λ .
Let X rec be an N -point independent sample from g and w rec :=
λ ( X rec ) .
Then, if( w quad , X quad ) is a proper kernel recombination of ( w rec , X rec ) for K 0 , it satisfies
[Equation]
where Z f := (cid:82) ℓ ( x ) f ( x ) d x and C K,f,g := (cid:82) K ( x, x ) λ ( x ) f ( x ) d x − (cid:82)(cid:82) K
( x, y ) f ( x ) f ( y ) d x d y .
The above approximation has one source of randomness which stems from sampling N points x recfrom g .
One can also apply this estimate with a random kernel and thereby introduce another sourceof randomness.
In particular, when we use the Nyström approximation for K 0 (that ensures K 1 is akernel [39]), then one can show that (cid:82) K 1 ( x, x ) f ( x ) d x can be bounded by
[Equation]
where σ n is the n -th eigenvalue of the integral operator L 2 ( f ) ∋ h (cid:55)→ (cid:82) K (
· , y ) h ( y ) f
( y )
d x ,
K max := sup x K ( x, x ) .
However, note that unlike Eq.
(12), this inequality only applies with highprobability due to the randomness of K 0 ; see Supplementary for details.
If, for example, K is a Gaussian kernel on R d and f is a Gaussian distribution, we have σ n =
O (exp( − cn 1 /d )) for some constant c >
(see Supplementary).
So in (12) we also achieve anempirically exponential rate when N ≫ C K,f,g .
RCHQ works well with a moderate M in practise.
Note that unlike the previous analysis [45], we do not have to assume that the space is compact.
Discussion
We introduced a batch BQ approach, BASQ, capable of simultaneous calculation of both modelevidence and posteriors.
BASQ demonstrated faster convergence (in wall-clock time) on bothsynthetic and real-world datasets, when compared against existing BQ approaches and state-of-the-art
NS.
Further, mathematical analysis shows the possibility to converge exponentially-fast under naturalassumptions.
As the BASQ framework is general-purpose, this can be applied to other active learning
GP-based applications, such as Bayesian optimisation [47], dynamic optimisation like control [22],and probabilistic numerics like ODE solvers [40].
Although it scales to the number of data seen inlarge-scale GP experiments, practical BASQ usage is limited to fewer than 16 dimensions (similar tomany GP-based algorithms).
However, RCHQ is agnostic to the input space, allowing quadrature inmanifold space.
An appropriate latent variable warped GP modelling, such as GPLVM [53], couldpave the way to high dimensional quadrature in future work.
In addition, while WSABI modelling limits the kernel to a squared exponential kernel, RCHQ allows to adopt other kernels or priorswithout a bespoke modelling BQ models.
(See Supplementary).
As for the mathematical proof, we do not incorporate batch and hyperparameter updates, which should be addressed in future work.
The generality of our theoretical guarantee with respect to kernel and distribution should be useful for extending the analysis to the whole algorithm.
Checklist
The checklist follows the references.
Please read the checklist guidelines carefully for information on how to 490 answer these questions.
For each question, change the default [TODO] to [Yes] , [No] , or [N/A] .
You arestrongly encouraged to include a justification to your answer , either by referencing the appropriate section
of
your paper
or
providing a
brief inline description.
For example
:• Did you include the license to the code and datasets?
[Yes] See Section ?? .•
Did you include the license to the code and datasets?
[No] The code and the data are proprietary.•
Did you include the license to the code and datasets?
[N/A]Please do not modify the questions and only use the provided macros for your answers.
Note that the Checklist 497 section does not count towards the page limit.
In your paper, please delete this instructions block and only keepthe Checklist section heading above along with the questions/answers below.1.
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes](b) Did you describe the limitations of your work?
[Yes] See section 7(c)
Did you discuss any potential negative societal impacts of your work?
[N/A](d)
Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes]2.
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes] see section 6 andSupplementary(b) Did you include complete proofs of all theoretical results?
[Yes] see section 6 and Supplementary3.
If you ran experiments...
[Table]
(a) Did you include the code, data, and instructions needed to reproduce the main experimentalresults (either in the supplemental material or as a URL)?
[Yes] in the supplemental material [Table] (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?[Yes] see section 5 and Supplementary
[Table]
(c) Did you report error bars (e.g., with respect to the random seed after running experimentsmultiple times)?
[Yes] see section
[Table]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs,internal cluster, or cloud provider)?
[Yes] see section
[Table]
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
[Table]
(a) If your work uses existing assets, did you cite the creators?
[Yes] see section 5(b)
Did you mention the license of the assets?
[Yes] see section
[Table]
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]
[Table]
(d) Did you discuss whether and how consent was obtained from people whose data you’re us-
[Table]
ing/curating?
[Yes] see section [Table]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
informa-
[Table]
tion or offensive content?
[N/A]
[Table]
If you used crowdsourcing or conducted research with human subjects...
[Table]
(a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[Table]
[N/A]
[Table]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) [Table] approvals, if applicable?
[N/A]
[Table]